Git Conversion Lazarus

This page has the detailed steps used to convert the Lazarus SVN repro to git.

SubGIT

edit

The initial conversion will be done using TMate Subgit.

Install

edit

Unpack subgit to

   ~/subgit/subgit-3.3.11

Required

edit

An author.txt file

 svn-name - git-name <email>

Run / main repro

edit
   cd ~/subgit/subgit-3.3.11
   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz.git
   gedit laz.git/subgit/config 

And add the following (the first line, replaces an existing setting)

   [svn]
     trunk = trunk:refs/heads/main
     gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
   [translate]  
     eols = false
     otherProperties = false
     ignores = false

The translate options prevent .gitignore and .gitattributes. Those can be translated, but will later be replaced...

   cp authors.txt laz.git/subgit/authors.txt
   ./bin/subgit install laz.git

Without skipping the gitignore/gitattributes, there will be an error at revision 47330. The process will hang. nterrupt it and edit

   gedit lazarus.git/svn/.metadata

and set branches-maxRev and tags-maxRev to 47332 (it should be at 47330) https://stackoverflow.com/questions/18918215/skip-revisions-when-installing-subgit-repository Continue with

   ./bin/subgit install laz-bin.git

Stop the daemon

   ./bin/subgit shutdown laz-bin.git/

Run / binaries repro

edit
   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus  laz-bin.git
   gedit laz-bin.git/subgit/config 
     [svn]
     gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
     trunk = binaries:refs/heads/main
     cp authors.txt laz-bin.git/subgit/authors.txt

follow the steps above

Run / website repro

edit
   ./bin/subgit configure https://svn.freepascal.org/svn/lazaruswebsite laz-web.git
   gedit laz-web.git/subgit/config 
     [svn]
     gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
     trunk = trunk:refs/heads/main
     cp authors.txt laz-web.git/subgit/authors.txt

follow the steps above

Part 2

edit
  • Updating gitignore
  • Prefixing all issue references with #
  • translating svn revisions to git hashes

Set a git user

edit
   git config --local user.email "lazarus@lazarus-ide.org"
   git config --local user.name "lazarus"

Clean out left overs from subgit

edit
   git for-each-ref --format="%(refname)" refs/svn/ | xargs -n 1 git update-ref -d

Clean left overs from binaries

edit

There are a few tags, that lead to binaries (gdb.exe) ending up in the data

  git tag -d binaries_0_9_26
  git tag -d binaries_0_9_26_2
  git tag -d binaries_0_9_28
  git tag -d binaries_0_9_30
  git tag -d binaries_0_9_30_2
  git tag -d binaries_0_9_30_2RC1
  git tag -d binaries_1_0

Updating gitignore/gitattributes / Prefix issues with #

edit

The below combines the 2 steps. It can be run in 2 separate calls.

It will run for about one hour.

   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --index-filter '  cp ~/subgit/subgit-3.3.11/.gitignore  ~/subgit/subgit-3.3.11/.gitattributes . ; git add .gitignore .gitattributes '  --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f  -- --all
   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch  --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f  -- --all

Cleanup

   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all ;git gc --prune=now

Translating references to svn rev (r12345) to #hash

edit

This must be the very last filter-branch. As each filter-branch creates new hashes for the commits, and that would invalidate the hash refs created here.

  • The first batch will update only commits it branch main.

This will also only be references to other commits in main.

Merges from branches to main can not be updated.

  • The 2nd batch will update only commits in branches. This means the hashes for the main branch will be left intact.

This will update merges from main to fixes

To verify the 2nd batch does not update the main branch compare the hash for git log -n1 main. It should not be changed by the 2nd batch.

   rm -rf /tmp/rmap
   mkdir /tmp/rmap
   git rev-list  main | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "git-svn-id: .*@[0-9][0-9]* "  | sed "s/.*@\([0-9]*\) .*/\1/" ) ; echo $0 > /tmp/rmap/$a'
   git rev-list  main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict; 
   unless ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print;  next; }
   my %f;  foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
     my \$rvl=\$1;    
     foreach my \$a (split / *[\/,] */, \$rvl) {
     unless (\$f{\$a}){
    \$f{\$a}=q/#/;
       my(\$a1, \$a2) = split / *- */, \$a;
    my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>;   chomp \$c; }
    my \$h; if(\$c && (open \$h, q/</, qq#../map/\$c#)) { 	  
       \$c=<\$h>;   chomp \$c;  \$c = q/#/.substr \$c, 0,10; 
         if (\$a2) {
        my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>;   chomp \$c2; }
        my \$h; if(\$c2 && (open \$h, q/</, qq#../map/\$c2#)) { 	  \$c2=<\$h>;   chomp \$c2;   	   
           \$c = \$c.q/-#/.substr( \$c2, 0,10); 
        }
        else { \$c = \$c.q/..# / unless \$c eq q/#/;  warn q/MAP2 FAIL /.\$c2 }
      }
      \$f{\$a}=\$c;
    }
    else {warn q/MAP1 fail/.\$a;}
     }} 
   }  
   foreach my \$k (keys %f) {   my \$v = \$f{\$k};
     unless (\$v eq q/#/) {     s/([ r,\/]\${k}\b)/\$1 \$v/g;   }
   }   
   warn \$_ if keys %f;
   print; "
   ' -f -- --all
   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   rm -rf /tmp/rmap
   mkdir /tmp/rmap
   git rev-list  --all | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "git-svn-id: .*@[0-9][0-9]* "  | sed "s/.*@\([0-9]*\) .*/\1/" ) ; echo $0 > /tmp/rmap/$a'
   git rev-list  main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict; 
   if ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print;  next; }
   my %f;  foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
     my \$rvl=\$1;    
     foreach my \$a (split / *[\/,] */, \$rvl) {
     unless (\$f{\$a}){
    \$f{\$a}=q/#/;
       my(\$a1, \$a2) = split / *- */, \$a;
    my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>;   chomp \$c; }
       if (\$c) {	unless ( -e qq#/tmp/rmap/m_\$c# ) { 
          my \$h; if(open \$h, q/</, qq#../map/\$c#) {   \$c=<\$h>;   chomp \$c; }
    }}
    if(\$c ) {
       \$c = q/#/.substr \$c, 0,10; 
         if (\$a2) {
        my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>;   chomp \$c2; }
           if (\$c2) {	unless ( -e qq#/tmp/rmap/m_\$c2# ) { 
              my \$h; if(open \$h, q/</, qq#../map/\$c2#) {   \$c2=<\$h>;   chomp \$c2; }
        }}
        if(\$c2) {   \$c = \$c.q/-#/.substr( \$c2, 0,10);     }
        else { \$c = \$c.q/..# / unless \$c eq q/#/;  warn q/MAP2 FAIL /.\$c2 }
      }
      \$f{\$a}=\$c;
    }
    else {warn q/MAP1 fail/.\$a;}
     }} 
   }  
   foreach my \$k (keys %f) {   my \$v = \$f{\$k};
     unless (\$v eq q/#/) {     s/([ r,\/]\${k}\b)/\$1 \$v/g;   }
   }   
   warn \$_ if keys %f;
   print; "
   ' -f -- --all
   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all ;git gc --prune=now
   rm -rf /tmp/rmap

Updating tags

edit

This must be done after all filter-branches

  • Annotate all tags.
  • Create some more tags

This needs the git user being set

   git for-each-ref --format="%(refname:short)" refs/tags/ | xargs -L1 /bin/sh -c 'git tag -f -a $0 -m $0 $0 '
   git rev-list --in-commit-order  --reverse   fixes_2_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_3" -m "main to version 2.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_2_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_1" -m "main to version 2.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_8..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_9" -m "main to version 1.9" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_6..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_7" -m "main to version 1.7" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_4..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_5" -m "main to version 1.5" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_3" -m "main to version 1.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_1" -m "main to version 1.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_30..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_31" -m "main to version 0.9.31" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_28..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_29" -m "main to version 0.9.29" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_26..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_27" -m "main to version 0.9.27" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_24..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_25" -m "main to version 0.9.25" $0'
   git rev-list --in-commit-order  --reverse   main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_0" -m "main init" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_2_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-2_2" -m "fixes 2.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_2_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-2_0" -m "fixes 2.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_8 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_8" -m "fixes 1.8 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_6 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_6" -m "fixes 1.6 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_4 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_4" -m "fixes 1.4 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_2" -m "fixes 1.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_0" -m "fixes 1.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_30 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_30" -m "fixes 0.9.30 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_28 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_28" -m "fixes 0.9.28 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_26 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_26" -m "fixes 0.9.26 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_24 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_24" -m "fixes 0.9.24 branch" $0'
   git rev-list --in-commit-order  --reverse   main..free-sparta | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-free-sparta" -m "free-sparta branch" $0'
   git rev-list --in-commit-order  --reverse   main..gtk-splitup | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-gtk-splitup" -m "gtk-splitup branch" $0'
   git rev-list --in-commit-order  --reverse   main..lcl-smartlink | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-lcl-smartlink" -m "lcl-smartlink branch" $0'
   git rev-list --in-commit-order  --reverse   main..listviewsortindicator | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-listviewsortindicator" -m "listviewsortindicator branch" $0'
   git rev-list --in-commit-order  --reverse   main..macosfullkeyboardaccess | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-macosfullkeyboardaccess" -m "macosfullkeyboardaccess branch" $0'

Finish

edit

edit HEAD to point to main

   gedit HEAD
   git remote add origin https://gitlab.com/freepascal.org/lazarus-ide/lazarus_test_conversion_X.git
   git push --all origin
   git push --tags origin

On gitlab

  • Settings / Repository
    • set default gitlab branch
      - no autoclose issues on default branch // decide later
    • set "reject unverified users"
    • protect branches * // no force push
    • protect tags: fixes* lazarus* release* main* // maintainers only
  • Settings / Integrations
    • External wiki
  • Settings / General
    • Disable Wiki, Snippets, Papges, Requirements, Operations, Service Desk, Container, Packages
      For now also CI, until we use it
    • Merge request approvals: Enable "Prevent users from modifying MR approval rules"
    • set the avatar
    • Add "description"
    • Make public
      • Disable "Users can request access"

Comparing data before/after filter-branch

edit

With subject line / Will show changes in subject

   git log --pretty='format:%ae %an %cd  %s %d' --graph --all > ../log.txt

Without subject line / Should have no diff

  • This compares the graph and committers => Thus it will show if all commits are present, and if all branches are correct
   git log --pretty='format:%ae %an %cd %d' --graph --all > ../log.txt
  • compare the changes to files (gitignore / gitattributes will diff for the top commit(s)) / --graph can be added too

(slow)

   git log --all --pretty='%ae %an %cd %d' --stat > ../statlog.txt
  • Compare the overall amount of lines in the log.
   git log --all | wc -l

Make sure you do that after "clean up" in each step. Otherwise the internal subgit or filter-branch revs will go into the compare.

Also "removing the binaries" introduces a diff.

Data

edit

Please mail me for any changes. Do not make changes here. Otherwise the changes will not make it into the conversion.

.gitignore

edit
   **/*.o
   **/*.ppu
   **/*.exe
   **/*.bak
   **/*.diff
   **/*.patch
   **/units/
   **/lib/
   **/*;[1-9]
   **/*.~*
   **/*.*~
   **/*.fpcunit.ini
   **/*.lps
   **/*.compiled
   /docs/chm/**/*.chm
   /docs/chm/**/*.xct
   /docs/chm/**/*.kwd
   /docs/chm/**/*.txt
   /libeay32.dll
   /ssleay32.dll
   /lazarus.cfg
   ide/revision.inc
   !**/patches/*.patch
   !**/patches/*.diff

.gitattributes

edit
   * text=auto !eol
   **/*.pp text
   **/*.pas text
   **/*.inc text
   **/*.lfm text
   **/*.lrs text
   **/*.lps text
   **/*.lpr text
   **/*.lpi text
   **/*.lpk text
   **/*.lpl text
   **/*.lrj text
   **/*.dpr text
   **/*.dfm text
   **/*.pot text
   **/*.po text
   **/*.rc text
   **/*.txt text
   **/*.txt.sample text
   **/*.ini text
   **/*.cfg text
   **/*.iss text
   **/*.isl text
   **/*.xml text
   **/*.xsl text
   **/*.html text
   **/*.css text
   **/*.php text
   **/*.js text
   **/.htaccess text
   **/*.patch text
   **/*.md text
   **/*.bat text
   **/*.sh text
   **/*.compiled text
   **/*.bdsproj text
   **/*.plist text
   **/*.h text
   **/*.c text
   **/*.cpp text
   **/*.in text
   **/*.xpm text
   **/*.svg text
   **/Makefile text
   **/Makefile.compiled text
   **/Makefile.fpc text
   **/*.exe -text
   **/*.dll -text
   **/*.so -text
   **/*.gif -text
   **/*.jpg -text
   **/*.ico -text
   **/*.bmp -text
   **/*.res -text
   **/*.ttf -text
   **/*.cur -text
   **/*.db3 -text
   **/*.dbf -text
   **/*.mbf -text
   **/*.mdx -text
   **/*.fbk -text
   **/*.odt -text
   **/*.ods -text
   **/*.odg -text
   **/*.odp -text
   **/*.xls -text
   **/*.pdf -text
   **/*.zip -text
   **/*.rar -text
   **/*.tar -text
   **/*.tgz -text
   **/*.gz -text
   **/*.icns -text
   **/*.dcr -text
   components/aggpas/**/*.ppm -text
   **/*.app/Contents/MacOS/* -text