You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by "Bruno P. Kinoshita" <ki...@apache.org> on 2021/05/08 06:15:46 UTC
Re: SIS web site repository (was: Report on current work)
Hi Martin,
I started porting the old history to Git to prepare a pull request to sis-site.
My first step was to list the users that committed to https://svn.apache.org/repos/asf/sis/site/trunk
```# https://docs.gitlab.com/ee/user/project/import/svn.html#cut-over-migration-with-svn2gitsvn log --quiet | grep -E "r[0-9]+ \| .+ \|" | cut -d'|' -f2 | sed 's/ //g' | sort | uniq
aestrada
ahart
desruisseaux
joes
jsorel
mattmann
pctony
rlaidlaw
smarru```
From this list, I think the following have GitHub users that we can map when importing the repo (see gitlab link above for more).
desruisseaux -> github.com/desruisseaux
jsorel -> github.com/jsorel (geoapi contributor)
mattmann -> github.com/chrismattmann
pctony -> github.com/pctony (ASF org)
rlaidlaw -> github.com/rlaidlaw (ASF org)
smarru -> github.com/smarru (ASF org)
But I don't know if the GitHub users match the SVN users for [aestrada, ahart, joes]. If we migrate with the SVN users, and they use different GitHub users, the commits of these users might be attributed to the wrong GitHub users.
I thought it would be best to check here before proceeding with the migration. What do you think we should do?
Cheers
Bruno
On Monday, 26 April 2021, 10:32:37 am NZST, Bruno P. Kinoshita <ki...@apache.org> wrote:
Hi Martin,
>Thanks. I have just created the Git repository at https://gitbox.apache.org/repos/asf/sis-site.git
Great news! And already mirrored at https://github.com/apache/sis-site
>Before to put the content rearranged for Hugo system, should we transfer the SVN history in that Git repository?
+1, sounds like a good plan. That way we retain the SCM history. Then prepare a pull request - or feel free to transfer the code from the PR over to that repository if you'd like.
After that we should be ready to start transferring the project over to git-scm.
Thanks Martin!
Bruno
On Sunday, 25 April 2021, 8:34:03 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Hello Bruno
Thanks. I have just created the Git repository at
https://gitbox.apache.org/repos/asf/sis-site.git
Before to put the content rearranged for Hugo system, should we transfer
the SVN history in that Git repository?
Martin
Le 25/04/2021 à 10:16, Bruno P. Kinoshita a écrit :
> I saw your e-mail, thanks. I've subscribed to the INFRA ticket, so
> once it's had any progress I'll be notified too, or ping me if you
> need anything (here, Slack, JIRA, etc).
>
Re: SIS web site repository
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
That's great news! Later when I have more spare time I will update the CSS libs (bootstrap I think) of both SIS site and GeoAPI, and try to improve accessibility.
Even though my eyes are not that bad, I do use a somewhat bigger screen font, so I sympathize a lot with visual impaired users :) and both sites are pretty easy to improve accessibility.
>Sorry for the extra work caused by my mistake
It was not a problem at all Martin, it's been fun working with you on SIS. And thanks a lot for going the extra mile and providing the command line examples with your message, that helped me understanding more your feedback :)
I'm still lurking in that INFRA ticket, so looking forward to the new site once they have finished migrating it.
Thanks!
Bruno
On Wednesday, 12 May 2021, 9:07:13 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Pull request merged
Just for the record, the "find" command that I provided in my previous
emails was wrong. It should have been:
find static/ -name "*.html" -exec git reset '{}' \;
Sorry for the extra work caused by my mistake
Martin
Re: SIS web site repository
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Pull request merged
Just for the record, the "find" command that I provided in my previous
emails was wrong. It should have been:
find static/ -name "*.html" -exec git reset '{}' \;
Sorry for the extra work caused by my mistake
Martin
Re: SIS web site repository
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Hello Hugo
Le 12/05/2021 à 00:58, Bruno P. Kinoshita a écrit :
> I tried to execute the commands you sent (thanks for that!), but it didn't work. Are you on macos? If so, macos' find has a different, non-posix, syntax I believe (had a similar issue with a co-worker some years ago and learned that, I think BSD introduced the changes).
Yes, for the "find" command MacOS requires that we specify the base
directory, which was ".". So the following command:
find -name "static/*.html" -exec git reset '{}' \;
become:
find . -name "static/*.html" -exec git reset '{}' \;
> Anyhow, I took the longer way, and edited each file that had http-https changes manually by looking at the GitHub UI to see what files had changes.
I'm sorry for the extra work!
> Also moved books/ to the top level of the Hugo site (won't be deployed, but it's available to devs as source to produce the new book I think?).
>
> Same with templates. I've moved the files from templates/ and content/templates both to under /templates/. It won't be in the SIS website, but devs can be use it.
>
> Let me know if it's looking better now, and if there are any changes pending.
>
> I've squashed the commits and did a quick test with Hugo again locally, and found no issues.
It all sound very good for me! At a first look I don't see anything to
change. I will push soon. Thanks again!
Martin
Re: SIS web site repository
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
Hi Martin,
I tried to execute the commands you sent (thanks for that!), but it didn't work. Are you on macos? If so, macos' find has a different, non-posix, syntax I believe (had a similar issue with a co-worker some years ago and learned that, I think BSD introduced the changes).
Anyhow, I took the longer way, and edited each file that had http-https changes manually by looking at the GitHub UI to see what files had changes.
Also moved books/ to the top level of the Hugo site (won't be deployed, but it's available to devs as source to produce the new book I think?).
Same with templates. I've moved the files from templates/ and content/templates both to under /templates/. It won't be in the SIS website, but devs can be use it.
Let me know if it's looking better now, and if there are any changes pending.
I've squashed the commits and did a quick test with Hugo again locally, and found no issues.
Cheers
Bruno
On Wednesday, 12 May 2021, 12:53:19 am NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Le 11/05/2021 à 13:36, Bruno P. Kinoshita a écrit :
> Out of curiosity, how long did `git svn clone` take? Just to compare
> with svn2git :)
>
I do not really know. It finished somewhere in the middle of the night.
But it has been many hours.
> I've created a PR here with the new site:
> https://github.com/apache/sis-site/pull/1
> Take a look to see if I forgot anything, or if something changed after
> the porting to Hugo.
>
Thanks! The main thing is the missing "book/*.html" files. Would it be
possible to add them, then squash the commits? With a squash, the pull
request will be automatically updated with a new "diff" page saying that
those files have been moved rather than deleted. The main question is
where to put them… Not in the "static" directory I think, since they are
source files used for building the static HTML file.
There is also many URL that changed. I think it is because I did an
update of those URLs after the migration to Hugo. Would it be possible
to do the following? Repeat the steps given in my previous email, but
before to do "git commit", execute the following commands:
find -name "static/*.html" -exec git reset '{}' \;
find -name "book/*" -exec git reset '{}' \;
git checkout .
It should cancel all changes in HTML files, on the assumption that
migration to Hugo should not change those files. Then the following
command would help to see if there is any remaining URL changes:
git diff --staged | grep "http"
Thanks!
Martin
Re: SIS web site repository
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Le 11/05/2021 à 13:36, Bruno P. Kinoshita a écrit :
> Out of curiosity, how long did `git svn clone` take? Just to compare
> with svn2git :)
>
I do not really know. It finished somewhere in the middle of the night.
But it has been many hours.
> I've created a PR here with the new site:
> https://github.com/apache/sis-site/pull/1
> Take a look to see if I forgot anything, or if something changed after
> the porting to Hugo.
>
Thanks! The main thing is the missing "book/*.html" files. Would it be
possible to add them, then squash the commits? With a squash, the pull
request will be automatically updated with a new "diff" page saying that
those files have been moved rather than deleted. The main question is
where to put them… Not in the "static" directory I think, since they are
source files used for building the static HTML file.
There is also many URL that changed. I think it is because I did an
update of those URLs after the migration to Hugo. Would it be possible
to do the following? Repeat the steps given in my previous email, but
before to do "git commit", execute the following commands:
find -name "static/*.html" -exec git reset '{}' \;
find -name "book/*" -exec git reset '{}' \;
git checkout .
It should cancel all changes in HTML files, on the assumption that
migration to Hugo should not change those files. Then the following
command would help to see if there is any remaining URL changes:
git diff --staged | grep "http"
Thanks!
Martin
Re: SIS web site repository
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
Hi Martin!
Out of curiosity, how long did `git svn clone` take? Just to compare with svn2git :)
I've created a PR here with the new site: https://github.com/apache/sis-site/pull/1
Take a look to see if I forgot anything, or if something changed after the porting to Hugo.
>However before to execute above instruction, I noticed that the "book"
directory seems to have disappeared. We would need to add it somewhere,
It should be in the static/book folder now. You will notice I left the templates folder outside, as I don't think they are intended to be published with the site, but rather used by developers for e-mails, releases, etc.
If you have archives that you want to include that will be accessible somewhere when the site is deployed, move them somewhere under that static folder. Otherwise use a top level folder. It shouldn't interefere with the ASF site publication, or how Hugo works.
Thanks heaps!
Bruno
On Tuesday, 11 May 2021, 10:36:25 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Hello Bruno and all
I have imported the SIS web site history with the following command:
git svn clone --stdlayout https://svn.apache.org/repos/asf/sis/site/trunk/ --authors-file=authors.txt
I pushed the result on the Git repository, mirrored there:
https://github.com/apache/sis-site
It contains the SVN revision numbers in each commit. I named the branch
"main" instead of "master" since it seems to be a recommended practice.
Le 10/05/2021 à 12:30, Bruno P. Kinoshita a écrit :
> Feel free to modify those commits, squash, edit. I don't mind even if
> the authorship changes too :)
>
I would like to keep the attribution to you if you don't mind :-). Can
you create a fork of above GitHub repository, then execute the following
Unix commands?
git clone https://github.com/path/to/your/repo/sis-site
git clone https://github.com/kinow/sis-site-draft
cd sis-site-draft
rm -rf .git
mv ../sis-site/.git .
git add -A
git commit -m "Some commit message"
git push
However before to execute above instruction, I noticed that the "book"
directory seems to have disappeared. We would need to add it somewhere,
I'm not sure where is the appropriate place. They are sources, but in
HTML instead of Markdown. After missing files are put in place, a "git
status" before above "git commit" should show "renamed" for almost every
files, except the ones that are specific to the CMS.
Martin
Re: SIS web site repository
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Hello Bruno and all
I have imported the SIS web site history with the following command:
git svn clone --stdlayout https://svn.apache.org/repos/asf/sis/site/trunk/ --authors-file=authors.txt
I pushed the result on the Git repository, mirrored there:
https://github.com/apache/sis-site
It contains the SVN revision numbers in each commit. I named the branch
"main" instead of "master" since it seems to be a recommended practice.
Le 10/05/2021 à 12:30, Bruno P. Kinoshita a écrit :
> Feel free to modify those commits, squash, edit. I don't mind even if
> the authorship changes too :)
>
I would like to keep the attribution to you if you don't mind :-). Can
you create a fork of above GitHub repository, then execute the following
Unix commands?
git clone https://github.com/path/to/your/repo/sis-site
git clone https://github.com/kinow/sis-site-draft
cd sis-site-draft
rm -rf .git
mv ../sis-site/.git .
git add -A
git commit -m "Some commit message"
git push
However before to execute above instruction, I noticed that the "book"
directory seems to have disappeared. We would need to add it somewhere,
I'm not sure where is the appropriate place. They are sources, but in
HTML instead of Markdown. After missing files are put in place, a "git
status" before above "git commit" should show "renamed" for almost every
files, except the ones that are specific to the CMS.
Martin
Re: SIS web site repository (was: Report on current work)
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
Hi Martin,
Feel free to modify those commits, squash, edit. I don't mind even if the authorship changes too :)
Let me know if you manage to move everything to apache/sis-site. I will do some quick testing then and we should be good to go!
Bruno
On Monday, 10 May 2021, 9:24:30 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Hello Bruno
Thanks a lot! I just cloned the repository. I have two observations:
I think it would be worth to squash commits from "Remove old site
contents" to "Use markdown instead of HTML for sources" inclusive. I
think that by doing so, the squashed commit would appear as directory
structure reorganization with relatively minor differences in file
content, instead of full replacement of all files.
But before doing so, I would be curious to give a try to "git svn" tools
because I think it preserves more metadata in commit messages, in
particular the SVN revision numbers of each commit. I'm giving a new try
to it right now.
Le 10/05/2021 à 06:00, Bruno P. Kinoshita a écrit :
> One issue that I found is that I cannot fork the apache/sis-site. I
> even tried using the GitHub REST API, but it simply returns an error
> saying I cannot fork empty repositories.
>
Let me try if I can put the site directory directly on my local clone of
apache/sis-site. I will post an update tonight or tomorrow.
Martin
Re: SIS web site repository (was: Report on current work)
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Hello Bruno
Thanks a lot! I just cloned the repository. I have two observations:
I think it would be worth to squash commits from "Remove old site
contents" to "Use markdown instead of HTML for sources" inclusive. I
think that by doing so, the squashed commit would appear as directory
structure reorganization with relatively minor differences in file
content, instead of full replacement of all files.
But before doing so, I would be curious to give a try to "git svn" tools
because I think it preserves more metadata in commit messages, in
particular the SVN revision numbers of each commit. I'm giving a new try
to it right now.
Le 10/05/2021 à 06:00, Bruno P. Kinoshita a écrit :
> One issue that I found is that I cannot fork the apache/sis-site. I
> even tried using the GitHub REST API, but it simply returns an error
> saying I cannot fork empty repositories.
>
Let me try if I can put the site directory directly on my local clone of
apache/sis-site. I will post an update tonight or tomorrow.
Martin
Re: SIS web site repository (was: Report on current work)
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
Hi Martin!
1.8 mi revisions in our ASF repository really slows down the migration. I executed the svn2git twice over the weekend, both time resulting in errors.
Then managed to import everything today. I started the process this morning, and it finished ~2PM, with the command: svn2git --rootistrunk --no-minimize-url https://svn.apache.org/repos/asf/sis/site/trunk/
I used my kinow/sis-site-draft repository to push the changes to. I pushed the `master` branch as-is. Then pushed a new commit removing the files from the old site.
And then I merged the `hugo` branch with `git merge --no-ff hugo`. You can see the results commits here: https://github.com/kinow/sis-site-draft/tree/master
Assuming we don't get any commits in our SVN repository, it should be good to go Martin.
One issue that I found is that I cannot fork the apache/sis-site. I even tried using the GitHub REST API, but it simply returns an error saying I cannot fork empty repositories.
I think we will need to either
- Get an initial commit with a LICENSE or README?
- Get an initial commit creating some dummy branch (not master? Nor hugo, to make the importing process easier?)
- Get an initial empty commit (--allow-empty); though not sure if that works, but could be the simplest? I reckon if we have an empty commit then `git rebase` should work?
Or happy to take any other suggestions. Once everything is in GitHub I will do one more quick test with the repository in apache/sis-site, then we can request INFRA to switch the site to git.
Thanks
Bruno
On Saturday, 8 May 2021, 10:31:38 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Le 08/05/2021 à 12:22, Bruno P. Kinoshita a écrit :
> Looks like ASF Infra thought about the authors issue too, they even
> provide a list of authors to use :)
>
I was not aware of that, nice finding!
> I left the process running here, but looks like it's going to take a
> really long time. Hopefully I can finish it by this weekend.
>
Yes, this is why after a few aborted attempts last week (I think I did
not provided correct "svn git" options) I thought to try on a smaller
scale first.
Thanks!
Martin
Re: SIS web site repository (was: Report on current work)
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Le 08/05/2021 à 12:22, Bruno P. Kinoshita a écrit :
> Looks like ASF Infra thought about the authors issue too, they even
> provide a list of authors to use :)
>
I was not aware of that, nice finding!
> I left the process running here, but looks like it's going to take a
> really long time. Hopefully I can finish it by this weekend.
>
Yes, this is why after a few aborted attempts last week (I think I did
not provided correct "svn git" options) I thought to try on a smaller
scale first.
Thanks!
Martin
Re: SIS web site repository (was: Report on current work)
Posted by "Bruno P. Kinoshita" <ki...@apache.org>.
Hi Martin,
>Thanks a lot! I started to play with "git svn" [1] recently for getting familiar with the process, but if you can do that, it would be greatly appreciated!
I realized I could import everything with git svn or another tool (I remember seeing one when Commons was migrating from SVN to Git too) and then simply rebase kinow/sis-site onto the imported sources.
And finally submit a pull request to the new repository. I should be able to complete it, but will shout if I need help :)
>For the list of authors, I think we can use the names and email addresses declared in the <developer> section of root pom.xml [2].
Looks like ASF Infra thought about the authors issue too, they even provide a list of authors to use :)
https://infra.apache.org/svn-to-git-migration.html
I left the process running here, but looks like it's going to take a really long time. Hopefully I can finish it by this weekend.
Thanks Martin
Bruno
On Saturday, 8 May 2021, 8:41:20 pm NZST, Martin Desruisseaux <ma...@geomatys.com> wrote:
Hello Hugo
Thanks a lot! I started to play with "git svn" [1] recently for getting
familiar with the process, but if you can do that, it would be greatly
appreciated!
For the list of authors, I think we can use the names and email
addresses declared in the <developer> section of root pom.xml [2].
Martin
[1] https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git
[2] https://github.com/apache/sis/blob/master/pom.xml#L123
Re: SIS web site repository (was: Report on current work)
Posted by Martin Desruisseaux <ma...@geomatys.com>.
Hello Hugo
Thanks a lot! I started to play with "git svn" [1] recently for getting
familiar with the process, but if you can do that, it would be greatly
appreciated!
For the list of authors, I think we can use the names and email
addresses declared in the <developer> section of root pom.xml [2].
Martin
[1] https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git
[2] https://github.com/apache/sis/blob/master/pom.xml#L123