You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Chandan Singh <ch...@chandansingh.net> on 2020/09/01 21:22:19 UTC
Re: MIgrating source code and CI - Summary
Hi all,
I think we've now made sufficient progress on our proof-of-concept that I am
confident in saying that I don't see any *technical* blockers to move to
GitHub.
That said, we will certainly need to take some actions, i.e. not all of these
steps can be automated. In this message, I wanted to summarize our inventory
and what I think should happen to each item. Since I don't have a good
experience with posting tables on mailing lists, I will go with the following
format:
# In case short-term and long-term plans are the same - the vast majority
ITEM // SHORT-TERM-PLAN
# In case plans are different for the short and the long terms
ITEM // SHORT-TERM-PLAN // LONG-TERM-PLAN
* Source code // automated migration
This one is simple - just one `git push -a` away.
* Issues // automated-migration
We can use the node-gitlab-2-github tool [0] that we tried earlier [1]. We
should be able to preserve the issue references, but all issues/comments will
be authored by the user that ran the migration tool.
* Merge requests // automated migration*
Merge requests can be migrated as pull requests using the same
node-gitlab-2-github tool. However there's a `*` here because references to
merge requests will NOT be preserved.
To clarify, it will not be the case that any references are pointing to an
unrelated issue. Rather, the `!` prefix is not considered "special" by GitHub
so it will just be rendered literally.
* Issue templates // manual migration
This should be pretty straightforward as well. In the worst case, there may
be a few small formatting differences between the two platforms. But I can't
imagine anything too bad.
* Contributor access // manual migration
All contributors who currently have access to our GitLab repository will need
to create a GitHub account (in case they don't have it already) and send one
of the maintainers a request to add them to our GitHub repository.
As far as CODEOWNERS feature is concerned, GitHub offers similar
functionality. We are using the bare minimal functionality from this file
anyway, so this shouldn't be a concern.
This is a bit of a tangent - if we go with the idea of using GitHub provided
runners for running pre-merge checks, we should also be able to change our CI
config such that we can accept PRs from forks. The reason we can't do this
today is because we run even the pre-merge checks on self-hosted runners.
* marge-bot // drop // use Actions
Our favorite marge-bot doesn't deal with GitHub pull requests as far as I
know. I would suggest to drop it for the time being, and have one of the
maintainers press the button manually.
And although this is not a blocker IMO, I think we should spend some time
writing a new GitHub "workflow" to mimic the "merge when CI passes"
functionality. Maybe some plugins already exist to support this.
Similar to how we organically arrived at marge-bot, I am assuming we will
find a natural solution after spending some time on GitHub. What do others
think?
* GitLab pages + artifacts // GitHub pages + artifacts
This is mostly relevant for our documentation website. We use GitLab pages
functionality to provide docs for `master` although it's not really
advertised - https://buildstream.gitlab.io/buildstream. What we do advertise
is the https://docs.buildstream.build/master website, that gets its
contents via public job artifacts of CI jobs.
For hosting the pages for master branch, there's an equivalent GitHub Pages
feature, and I'm sure someone would have written a plugin to push contents of
a job to the `gh-pages` branch.
For the artifacts, we have the equivalent GitHub artifacts [2] feature.
* CI // manual migration*
This is by far the most involved task compared to the rest. But, we have a
head start here. Recently Chris has contributed some changes that bring us
closer to having parity with the current GitLab CI setup.
There's a `*` here because of the following caveats:
1. We have decided to drop the WSL jobs [3] so they will not be migrated.
2. As we have found out, the GitHub runners aren't powerful enough to be able
to run the overnight tests in a reasonable time. So, I would like to
request the current maintainers of our CI runners to consider providing
similar runners for the overnight tests on GitHub. As I understand it,
this would essentially involve setting up the GitHub Actions Runner [4]
on those machines.
On the bright side, we only need self-hosted runners for more resource
intensive jobs like building the freedesktop project, but not pre-merge
checks. So, in theory, the digitalocean bill should go down :)
--------------
I think that's all the things that I can think of that will need to be
migrated. Please respond if you think I've missed something or if you disagree
with the proposed alternatives.
Let me know what you think!
Cheers,
Chandan
[0]: https://github.com/piceaTech/node-gitlab-2-github
[1]: https://lists.apache.org/thread.html/r99ad14961ec85b05b532e195c4c46737a3904d51015c12660c7fc93f%40%3Cdev.buildstream.apache.org%3E
[2]: https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts
[3]: https://lists.apache.org/thread.html/rc49b6416980093ae9e73fda923e43ed668daa0c2a8d1bc94e52b774f%40%3Cdev.buildstream.apache.org%3E
[4]: https://github.com/actions/runner
Re: MIgrating source code and CI - Summary
Posted by Chandan Singh <ch...@chandansingh.net>.
Hi Jürg,
> On Tue, 2020-09-01 at 22:22 +0100, Chandan Singh wrote:
> > * Issues // automated-migration
> >
> > We can use the node-gitlab-2-github tool [0] that we tried earlier [1]. We
> > should be able to preserve the issue references, but all issues/comments will
> > be authored by the user that ran the migration tool.
>
> We should create a neutral GitHub account for the import to reduce
> confusion with regards to issue and comment authorship.
Definitely +1 for a separate account.
> The issue numbers stay the same, correct? I.e., not only references
> between issues but also issue references in commit messages and mails
> will be preserved.
That's correct, issue numbering is preserved and the references should
work fine from commits etc as well. With the obvious caveat that the
references would only go to the GitHub issue if the short form
(#number) was used, and not if a full issue link was used.
> > * Merge requests // automated migration*
> >
> > Merge requests can be migrated as pull requests using the same
> > node-gitlab-2-github tool. However there's a `*` here because references to
> > merge requests will NOT be preserved.
> >
> > To clarify, it will not be the case that any references are pointing to an
> > unrelated issue. Rather, the `!` prefix is not considered "special" by GitHub
> > so it will just be rendered literally.
>
> As far as I know, the merge requests will be renumbered (unlike
> issues). I understand that `!` MR references will not automatically be
> replaced. However, can we at least provide developers a way to manually
> lookup the new GitHub PR number from a GitLab MR number? Could simply
> be a table of GL MR -> GH PR mappings. Or could we add the old !MR
> number to the title of each migrated GitHub PR?
I think such a table might be useful as it'll work both ways. This
table could then also be used to add a comment if we want that.
I can't remember now, but I think node-gitlab-2-github outputs this
info as part of its logs. Even if it doesn't, it should still be easy
enough to create such a table, as the numbers should be increasing
monotonically on both sides. (node-gitlab-2-github will create all
issues first, and then the PRs)
> > * marge-bot // drop // use Actions
> >
> > Our favorite marge-bot doesn't deal with GitHub pull requests as far as I
> > know. I would suggest to drop it for the time being, and have one of the
> > maintainers press the button manually.
> >
> > And although this is not a blocker IMO, I think we should spend some time
> > writing a new GitHub "workflow" to mimic the "merge when CI passes"
> > functionality. Maybe some plugins already exist to support this.
>
> Might https://github.com/apps/mergewhenready be useful?
Quite likely. That's something that came up in my searches as well. I
don't have first-hand experience of using this myself but we'll learn
as we go along I guess :)
> What about the merge method? On GitLab we use 'Merge commit with semi-
> linear history'. Does GitHub support this already?
Good question! I think the answer is yes. Although this functionality
comes in form of two different knobs in GitHub that we can tweak:
1. When we set up branch protection rules, we will have to mark
certain status checks as required (presumably all the current
pre-merge checks). While doing so, we can enable the "Require branches
to be up to date before merging" checkbox. This will give us the
semi-linear part.
2. About the merge method itself, we have a choice between "merge",
"squash & merge" and "rebase & merge". Assuming we enable the above
check, the "merge" strategy would be equivalent to what we're doing
now.
Cheers,
Chandan
Re: MIgrating source code and CI - Summary
Posted by Jürg Billeter <j...@bitron.ch>.
Hi Chandan,
Thanks for writing this up.
On Tue, 2020-09-01 at 22:22 +0100, Chandan Singh wrote:
> * Issues // automated-migration
>
> We can use the node-gitlab-2-github tool [0] that we tried earlier [1]. We
> should be able to preserve the issue references, but all issues/comments will
> be authored by the user that ran the migration tool.
We should create a neutral GitHub account for the import to reduce
confusion with regards to issue and comment authorship.
The issue numbers stay the same, correct? I.e., not only references
between issues but also issue references in commit messages and mails
will be preserved.
> * Merge requests // automated migration*
>
> Merge requests can be migrated as pull requests using the same
> node-gitlab-2-github tool. However there's a `*` here because references to
> merge requests will NOT be preserved.
>
> To clarify, it will not be the case that any references are pointing to an
> unrelated issue. Rather, the `!` prefix is not considered "special" by GitHub
> so it will just be rendered literally.
As far as I know, the merge requests will be renumbered (unlike
issues). I understand that `!` MR references will not automatically be
replaced. However, can we at least provide developers a way to manually
lookup the new GitHub PR number from a GitLab MR number? Could simply
be a table of GL MR -> GH PR mappings. Or could we add the old !MR
number to the title of each migrated GitHub PR?
I assume we won't delete the BuildStream project on GitLab and rather
make it read-only/archive (with a link to GitHub), so it would be
possible to find the MR title via the read-only GitLab project.
However, it would be good if we didn't depend on this.
We could likely also extract a table of GitLab MR number to title from
a GitLab project export as a simple stopgap.
> * marge-bot // drop // use Actions
>
> Our favorite marge-bot doesn't deal with GitHub pull requests as far as I
> know. I would suggest to drop it for the time being, and have one of the
> maintainers press the button manually.
>
> And although this is not a blocker IMO, I think we should spend some time
> writing a new GitHub "workflow" to mimic the "merge when CI passes"
> functionality. Maybe some plugins already exist to support this.
Might https://github.com/apps/mergewhenready be useful?
What about the merge method? On GitLab we use 'Merge commit with semi-
linear history'. Does GitHub support this already?
Cheers,
Jürg