You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Tristan Van Berkom <tr...@codethink.co.uk> on 2020/12/28 10:21:53 UTC
GitHub migration
Hi all,
As we have known for some time now, the BuildStream project will be
moving to GitHub.
Since we are in the holiday season and we are confident that it will
not be disruptive, we are going to do our best to push this over the
line during this lull, and kick off 2021 development using a github
workflow.
Migration plan
--------------
Chandan made a lot of progress some months ago to get CI mostly working
in github[0].
Tom Pollard, Douglas Winship, Chris Polin and I took another crack at
it last month and we've ironed out most of the creases to our
satisfaction.
This mostly includes solving the release process and the problem
of automatically publishing documentation of prior releases.
We created a writeup of our plan to perform the migration here[1].
What are we migrating ?
-----------------------
We plan to migrate the following repositories to the apache namespace
in github, at https://github.com/apache/.
* The main BuildStream core repository
https://gitlab.com/BuildStream/buildstream/
* The repository which generates the website
https://gitlab.com/BuildStream/website/
* The repository which aggregates and publishes docs for released
versions of BuildStream
https://gitlab.com/BuildStream/docs-website/
There are no plans to migrate the plugin repositories at this time.
While we may end up creating some new plugin specific repositories on
github in the future, we expect mainly to continue to use gitlab for
plugin repositories in the long term, this will be easier for plugin
maintainers to release asynchronously from core BuildStream and with a
bit less bureaucracy.
BuildStream first
-----------------
We have recently acquired https://github.com/apache/buildstream/ for
our purposes to migrate the core BuildStream repository.
All three repositories will not be migrated in one go, BuildStream will
be migrated first.
Loss
----
Things may be a bit shaky for a time, there will be some minor loss in
the migration, but we've tested the migration and we're confident that
loss is minimal.
For example, some things like already merged/closed merge requests may
not be migrated as pull requests, and comments and issues will not
automatically be assigned to your corresponding username on github.
Thanks to Douglas's scripting we have managed to preserve references to
gitlab usernames referred to in comments: links to users in historical
issues and merge requests will refer to users on gitlab, and not users
which might not exist on github, or might exist but not be the same
person.
Some of the CI will have to be revived post migration, mostly this is
limited to remote execution testing - we have agreed to address this
after the migration.
Summary
-------
With any luck, we will get this done swiftly by the time people start
coming back to work in 2021, and I hope to see you all next year on
github :)
I will be periodically replying to this thread to announce timeframes
as I coordinate with Apache folks and sort out permissions and such.
And of course, Merry Christmas and Happy New Year !!
Cheers,
-Tristan
[0]: https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
[1]: https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
On Sun, 2021-01-10 at 21:19 +0000, Javier Jardón wrote:
> On 2021-01-10 09:28, Tristan Van Berkom wrote:
> >
[...]
> > This was fixed by running CI on our repo on the `pull_request` event,
> > which will be triggered in our repo whenever a pull request is created
> > even if it is created from a downstream fork.
> >
> > It is a well known issue that running CI on both "push" events and
> > "pull_request" events causes CI to run twice, which is wasteful,
> > however we do want the ability to run CI in advance of creating a pull
> > request.
> >
> > While github has yet to provide a solution for this issue, I was able
> > to find this workaround[3], testing so far shows this is working as
> > expected.
>
> This doesn't seem to work on this PR (generated from a fork, rebased
> against current master): [1]
Indeed, this PR also does not run the actions for a pull request
originating from a forked repo:
https://github.com/apache/buildstream/pull/1440
Even though it is working well on the testbed:
https://github.com/buildstream-migration/buildstream/pull/13
This appears to be more restrictions from the apache namespace on
github.
/me bangs head on desk.
Cheers,
-Tristan
Re: GitHub migration
Posted by Javier Jardón <ja...@codethink.co.uk>.
On 2021-01-10 09:28, Tristan Van Berkom wrote:
> Hi all,
Hi Tristan,
Thanks for the updates
<snip>
>> * Status checks for CI which is running on pull requests don't seem
>> to appear in the pull request UI.
>>
>> This may be only an issue for pull requests submitted from "forked"
>> repositories.
>>
>> Since we expect most contributions to come from "forked" repos on
>> github, it would be good to fix this (and from prior experience,
>> I've
>> seen this work properly before on other projects).
>>
>> E.g.: At the bottom of the page here:
>>
>> https://github.com/apache/buildstream/pull/1436
>>
>> We can see some status about the PR, but it is missing the status
>> from "workflow actions".
>
> This was fixed by running CI on our repo on the `pull_request` event,
> which will be triggered in our repo whenever a pull request is created
> even if it is created from a downstream fork.
>
> It is a well known issue that running CI on both "push" events and
> "pull_request" events causes CI to run twice, which is wasteful,
> however we do want the ability to run CI in advance of creating a pull
> request.
>
> While github has yet to provide a solution for this issue, I was able
> to find this workaround[3], testing so far shows this is working as
> expected.
This doesn't seem to work on this PR (generated from a fork, rebased
against current master): [1]
Cheers,
Javier
[1] https://github.com/apache/buildstream/pull/1436
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
After some discussion, we have concluded that we would much prefer to
perform the gitlab data migration again from scratch than to live with
the off-by-one issue.
Daniel, I've added you on CC to this to humbly request assistance in
this matter.
Essentially, as described in the quoted text below, the migration
missed one issue and has resulted in 600+ issue numbers being off by
one, which is quite a hassle (references to issues in many commit
messages and migrated pull requests are incorrect, etc).
On Sun, 2021-01-10 at 18:28 +0900, Tristan Van Berkom wrote:
> Hi all,
[...]
> All migrated issues up until #800 have the same issue number in gitlab
> and in github, however issue #801 was skipped in the migration, due to
> the migration script believing that #801 was in fact the already
> migrated #423.
>
> The migration script avoids accidentally migrating the same record
> twice, probably in case the script needed to be restarted (which is
> did):
>
> https://github.com/Cynical-Optimist/node-gitlab-2-github/blob/master/src/index.ts#L281
>
> The net result, we do not have an issue #801 in github, and every
> subsequent following issue in github is numbered "${gitlab_issue} - 1".
[...]
What we would need help with would be much like what we went through
over christmas (but now we know all the ins and outs of it so it would
run more smoothly):
* Delete the repo
* Recreate the bare repo
* Give the migration user temporary write access to the repo:
https://github.com/BuildStream-Migration-Bot
* Enable the issues and pull requests features on the repo
(issue tracking is disabled by default in the ASF repos)
* Temporarily disable automated emails during the migration (asf
infra has hooks which sends emails for new issues and pull requests
to the mailing list).
* Set the default branch to `master`
To avoid the issue, I would modify the title of #801 in gitlab so that
it does not get mistaken for #423 and rerun the process.
Would infra be able to assist us with this ?
Apologies for the hassle, we really should have caught this issue
before in our test runs but the problem went unnoticed until this
weekend.
Cheers,
-Tristan
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
Here are some status updates.
TLDR:
- There is an issue with the migration causing over half of our
issues to be off-by-one issue numbers.
- Further investigation on branch protection shows we cannot get
a suitable setup using .asf.yaml, we will need support.
- Feedback and CI on merge requests originating from forks is
now fixed.
First, the migration issue.
All migrated issues up until #800 have the same issue number in gitlab
and in github, however issue #801 was skipped in the migration, due to
the migration script believing that #801 was in fact the already
migrated #423.
The migration script avoids accidentally migrating the same record
twice, probably in case the script needed to be restarted (which is
did):
https://github.com/Cynical-Optimist/node-gitlab-2-github/blob/master/src/index.ts#L281
The net result, we do not have an issue #801 in github, and every
subsequent following issue in github is numbered "${gitlab_issue} - 1".
This is not the end of the world, but it is quite bad. Most notably, we
need to pay extra attention when merging pull requests which have been
open pre-migration, and check that commit messages and PR comments are
amended to refer to the correct issue number.
On Thu, 2021-01-07 at 22:20 +0900, Tristan Van Berkom wrote:
> Hi all,
[...]
> Todo list
> ---------
>
> * Configuration of protected branches.
>
> This is a little complicated without access to the repository
> configuration UI (I managed to temporarily lock myself out during
> the migration).
>
> The most important part will be to ensure we dont allow any
> non-fastforward merges on master or any release branch, this is
> rather high priority.
I ran some tests today updating the relevant portions of .asf.yaml[0],
and it seems there is no way to properly set this up. We will certainly
need help from the infra team to get this sorted.
I have added the relevant CODEOWNERS file, which does appear to work,
but there is no way to allow self-approval of pull requests on
protected branches[1], so I think that requiring code-owner reviews is
unworkable for us.
Disallowing non-fastforward merges / force pushes on protected branches
is also just not working.
The "Prevent force pushes" section[0] says that specifying a single
branch without options should cause this, but I was unable to reproduce
that behavior, nor do I understand how the .asf.yaml handling code[2]
would cause that to happen, I suspect the wiki is outdated (also, we
need to protect multiple branches, not just master).
I have tried a simpler setup by setting "strict: True" on multiple
branches, without requiring review, like so:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Testing basic protection of multiple branches,
# this needs to be tested on the master branch.
#
protected_branches:
# Test 1
#
tristan/test-protected-branch-1:
required_status_checks:
# strict means "Require branches to be up to date before merging".
strict: true
# Test 2
#
tristan/test-protected-branch-2:
required_status_checks:
# strict means "Require branches to be up to date before merging".
strict: true
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I received automated emails informing me that:
"GitHub Protected Branches has been enabled on branch=tristan/test-protected-branch-1"
However, subsequently:
I was able to merge this pull request without rebasing:
https://github.com/apache/buildstream/pull/1437
A second try shows I am still allowed to merge a pull request
without rebasing against the target branch:
https://github.com/apache/buildstream/pull/1439
> * Status checks for CI which is running on pull requests don't seem
> to appear in the pull request UI.
>
> This may be only an issue for pull requests submitted from "forked"
> repositories.
>
> Since we expect most contributions to come from "forked" repos on
> github, it would be good to fix this (and from prior experience, I've
> seen this work properly before on other projects).
>
> E.g.: At the bottom of the page here:
>
> https://github.com/apache/buildstream/pull/1436
>
> We can see some status about the PR, but it is missing the status
> from "workflow actions".
This was fixed by running CI on our repo on the `pull_request` event,
which will be triggered in our repo whenever a pull request is created
even if it is created from a downstream fork.
It is a well known issue that running CI on both "push" events and
"pull_request" events causes CI to run twice, which is wasteful,
however we do want the ability to run CI in advance of creating a pull
request.
While github has yet to provide a solution for this issue, I was able
to find this workaround[3], testing so far shows this is working as
expected.
Cheers,
-Tristan
[0]: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features#git.asf.yamlfeatures-BranchProtection
[1]: https://github.community/t/do-not-require-owner-approval-if-the-pull-request-is-from-an-owner/369
[2]: https://github.com/apache/infrastructure-puppet/blob/deployment/modules/gitbox/files/asfgit/asfyaml.py#L376
[3]: https://github.community/t/how-to-trigger-an-action-on-push-or-pull-request-but-not-both/16662/10
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
Most everything I could get done without blocking on external forces
has been done now, and it's time for the status report.
Completed actions
-----------------
* The buildstream repository has been migrated successfully to:
https://github.com/apache/buildstream/
This includes merge requests and issues, barring some minor
loss as outlined in the migration wiki page[0]
* The releases have been setup in the new github repository:
https://github.com/apache/buildstream/releases
All documentation packages from prior releases have been carried
over from historical builds on gitlab, and attached to the releases
on github.
The release announcement emails for every historical release have
also been linked to every release on github.
* The base pre/post merge CI has been setup on github - tests are run
in pre merge CI and bleeding edge master documentation is published
whenever merging something to master.
This excludes:
- Overnight tests
- Remote execution tests
- Windows platform (partial) tests
* References to gitlab have been replaced with references to github.
This includes:
- The buildstream repository
- The docs-website repository
- The website repository
* The docs-website documentation aggregation has been updated.
This repository now obtains the documentation packages for the
releases it decides to publish documentation for from github releases
instead of gitlab pipeline artifacts.
Todo list
---------
* Configuration of protected branches.
This is a little complicated without access to the repository
configuration UI (I managed to temporarily lock myself out during
the migration).
The most important part will be to ensure we dont allow any
non-fastforward merges on master or any release branch, this is
rather high priority.
* Recover lost CI
Some work and thought needs to be put into how we can revive some
of our lost CI:
- Overnight tests
These take too long for standard github runners, can we get support
from Apache infra to run the heavy duty overnights ?
- Remote execution tests
This is just complicated to achieve in github, requires work.
- Windows platform (partial) tests
This needs further investigation.
* Status checks for CI which is running on pull requests don't seem
to appear in the pull request UI.
This may be only an issue for pull requests submitted from "forked"
repositories.
Since we expect most contributions to come from "forked" repos on
github, it would be good to fix this (and from prior experience, I've
seen this work properly before on other projects).
E.g.: At the bottom of the page here:
https://github.com/apache/buildstream/pull/1436
We can see some status about the PR, but it is missing the status
from "workflow actions".
* The github "pages" needs to be properly enabled.
We're facing some minor technical difficulty here, this was tested
to work in the test migration but needs assistance from Apache infra
team to setup properly on github.
The documentation is being published to our `gh-pages` branch
automatically but it is not appearing yet at:
https://buildstream.apache.github.io/buildstream/
* Badges and links for latest releases and snapshots need to be fixed.
These will be fixed once github pages is fixed, as the "latest build"
of master documentation is the single source of truth for these
badges.
* The docs-website currently excludes documentation for the master
branch.
Again, this will be fixed once github pages is sorted.
* The code coverage support in the github UI is not enabled yet.
This requires some support from Apache infra, the "codecov"
github application installed on apache github
(https://codecov.io/gh/apache) needs to be given the proper
permission to the new buildstream repository.
Once that is done, I can merge Tom Pollards work:
https://github.com/buildstream-migration/buildstream/pull/8
And it should provide us with a beautiful code coverage UX
(somewhat better than what we had with gitlab, even).
* Documentation updates
While updating the websites and docs I noticed there are some
parts which were not easy to update.
Mostly:
- Instructions for contributors on how to contribute patches
- Instructions for maintainers concerning the release process
These bits of documentation are mostly in CONTRIBUTING.rst and
need to be updated once we have figured out our workflow a bit
more clearly.
* Repositories still need to be migrated
The following repos:
- https://gitlab.com/BuildStream/website/
- https://gitlab.com/BuildStream/docs-website/
Are to be migrated to github too.
This entails:
- Obtaining new github repositories under the apache namespace
- Implementing the CI for building and publishing these docs in
github (this is already done in the corresponding tested
migrated repositories https://github.com/buildstream-migration/)
- Updating the CNAME records for the dns to resolve to the new
locations.
Looking forward
---------------
Not strictly as a part of this migration, but related:
* Cache server used for overnight tests
We are currently hosting a testing CAS service independently of the
tests, mostly for the purpose of the overnight tests.
What should we do about this ? Shut it down ?
Move this service to apache infra in some place that we can easily
update the service and redeploy it without much hassle ?
* Move to freenode IRC
We're currently looking into how we can get proper ownership of our
own #buildstream channel on freenode, once that is done we will
announce and move our IRC to freenode.
At this point, we will try to update all documentation pointing to
our IRC channel.
As agreed upon late last year, will will update this to equally
promote the use of "slack".
Cheers,
-Tristan
[0]: https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
On Sat, 2021-01-02 at 19:33 +0900, Tristan Van Berkom wrote:
> Hi all, and happy new year !
>
> The migration script has completed and I have now locked down the
> BuildStream project on gitlab, setting it to archive mode, and
> replacing the main README.rst with a notice and link to the new
> repository on GitHub:
>
> https://github.com/apache/buildstream/
>
> There is still some follow up work to do, such as migrating the
> historical release data and docs builds, and setting up the CI and
> review process on GitHub properly.
>
> I will report back to the list when things are good to go.
>
> Cheers,
> -Tristan
>
> On Wed, 2020-12-30 at 20:40 +0900, Tristan Van Berkom wrote:
> > Hi again,
> >
> > On Wed, 2020-12-30 at 19:34 +0900, Tristan Van Berkom wrote:
> > > Hi all,
> > >
> > > As expected, something always goes wrong.
> > >
> > > You will have received a lot of spam mail from the dev mailing
> > > list now
> > > for the issue migration of only the first 20 issues and their
> > > comments,
> > > so I have temporarily halted the migration until I can sort this.
> > >
> >
> > Crisis averted; we were able to disable the notifications and the
> > migration is chugging along smoothly and silently.
> >
> > Cheers,
> > -Tristan
> >
> > > Cheers,
> > > -Tristan
> > >
> > >
> > > On Wed, 2020-12-30 at 18:40 +0900, Tristan Van Berkom wrote:
> > > > Hi all,
> > > >
> > > > I am about to run the migration script which will migrate issue
> > > > and
> > > > merge request and such records from gitlab to github.
> > > >
> > > > This is expected to take over 12 hours to complete, and is not
> > > > expected
> > > > to generate immense spam after having tested this before and
> > > > making
> > > > adjustments to the migration script.
> > > >
> > > > That said, things can always go wrong, as this is a one-off it
> > > > will
> > > > be
> > > > best to let the script complete.
> > > >
> > > > I will not be setting the gitlab repository into a read-only
> > > > archived
> > > > state until after the migration completes, as this is untested,
> > > > so:
> > > >
> > > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > > !!! !!!
> > > > !!! Any new issues, merge requests, or comments !!!
> > > > !!! made to gitlab after this point may very well !!!
> > > > !!! be lost. !!!
> > > > !!! !!!
> > > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > >
> > > > Please refrain from updating the buildstream repo in gitlab
> > > > during
> > > > this
> > > > time.
> > > >
> > > > This has been tested a few times already, but of course many
> > > > things
> > > > can
> > > > go wrong, fingers crossed !
> > > >
> > > > Cheers,
> > > > -Tristan
> > > >
> > > > On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> > > > > Hi all,
> > > > >
> > > > > As we have known for some time now, the BuildStream project
> > > > > will
> > > > > be
> > > > > moving to GitHub.
> > > > >
> > > > > Since we are in the holiday season and we are confident that
> > > > > it
> > > > > will
> > > > > not be disruptive, we are going to do our best to push this
> > > > > over
> > > > > the
> > > > > line during this lull, and kick off 2021 development using a
> > > > > github
> > > > > workflow.
> > > > >
> > > > > Migration plan
> > > > > --------------
> > > > > Chandan made a lot of progress some months ago to get CI
> > > > > mostly
> > > > > working
> > > > > in github[0].
> > > > >
> > > > > Tom Pollard, Douglas Winship, Chris Polin and I took another
> > > > > crack
> > > > > at
> > > > > it last month and we've ironed out most of the creases to our
> > > > > satisfaction.
> > > > >
> > > > > This mostly includes solving the release process and the
> > > > > problem
> > > > > of automatically publishing documentation of prior releases.
> > > > >
> > > > > We created a writeup of our plan to perform the migration
> > > > > here[1].
> > > > >
> > > > >
> > > > > What are we migrating ?
> > > > > -----------------------
> > > > > We plan to migrate the following repositories to the apache
> > > > > namespace
> > > > > in github, at https://github.com/apache/.
> > > > >
> > > > > * The main BuildStream core repository
> > > > > https://gitlab.com/BuildStream/buildstream/
> > > > >
> > > > > * The repository which generates the website
> > > > > https://gitlab.com/BuildStream/website/
> > > > >
> > > > > * The repository which aggregates and publishes docs for
> > > > > released
> > > > > versions of BuildStream
> > > > > https://gitlab.com/BuildStream/docs-website/
> > > > >
> > > > > There are no plans to migrate the plugin repositories at this
> > > > > time.
> > > > >
> > > > > While we may end up creating some new plugin specific
> > > > > repositories
> > > > > on
> > > > > github in the future, we expect mainly to continue to use
> > > > > gitlab
> > > > > for
> > > > > plugin repositories in the long term, this will be easier for
> > > > > plugin
> > > > > maintainers to release asynchronously from core BuildStream
> > > > > and
> > > > > with a
> > > > > bit less bureaucracy.
> > > > >
> > > > >
> > > > > BuildStream first
> > > > > -----------------
> > > > > We have recently acquired
> > > > > https://github.com/apache/buildstream/
> > > > > for
> > > > > our purposes to migrate the core BuildStream repository.
> > > > >
> > > > > All three repositories will not be migrated in one go,
> > > > > BuildStream
> > > > > will
> > > > > be migrated first.
> > > > >
> > > > >
> > > > > Loss
> > > > > ----
> > > > > Things may be a bit shaky for a time, there will be some
> > > > > minor
> > > > > loss
> > > > > in
> > > > > the migration, but we've tested the migration and we're
> > > > > confident
> > > > > that
> > > > > loss is minimal.
> > > > >
> > > > > For example, some things like already merged/closed merge
> > > > > requests
> > > > > may
> > > > > not be migrated as pull requests, and comments and issues
> > > > > will
> > > > > not
> > > > > automatically be assigned to your corresponding username on
> > > > > github.
> > > > >
> > > > > Thanks to Douglas's scripting we have managed to preserve
> > > > > references to
> > > > > gitlab usernames referred to in comments: links to users in
> > > > > historical
> > > > > issues and merge requests will refer to users on gitlab, and
> > > > > not
> > > > > users
> > > > > which might not exist on github, or might exist but not be
> > > > > the
> > > > > same
> > > > > person.
> > > > >
> > > > > Some of the CI will have to be revived post migration, mostly
> > > > > this
> > > > > is
> > > > > limited to remote execution testing - we have agreed to
> > > > > address
> > > > > this
> > > > > after the migration.
> > > > >
> > > > >
> > > > > Summary
> > > > > -------
> > > > > With any luck, we will get this done swiftly by the time
> > > > > people
> > > > > start
> > > > > coming back to work in 2021, and I hope to see you all next
> > > > > year
> > > > > on
> > > > > github :)
> > > > >
> > > > > I will be periodically replying to this thread to announce
> > > > > timeframes
> > > > > as I coordinate with Apache folks and sort out permissions
> > > > > and
> > > > > such.
> > > > >
> > > > > And of course, Merry Christmas and Happy New Year !!
> > > > >
> > > > > Cheers,
> > > > > -Tristan
> > > > >
> > > > >
> > > > > [0]:
> > > > > https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> > > > > [1]:
> > > > > https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
> > > > >
> > > > >
> > > > >
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all, and happy new year !
The migration script has completed and I have now locked down the
BuildStream project on gitlab, setting it to archive mode, and
replacing the main README.rst with a notice and link to the new
repository on GitHub:
https://github.com/apache/buildstream/
There is still some follow up work to do, such as migrating the
historical release data and docs builds, and setting up the CI and
review process on GitHub properly.
I will report back to the list when things are good to go.
Cheers,
-Tristan
On Wed, 2020-12-30 at 20:40 +0900, Tristan Van Berkom wrote:
> Hi again,
>
> On Wed, 2020-12-30 at 19:34 +0900, Tristan Van Berkom wrote:
> > Hi all,
> >
> > As expected, something always goes wrong.
> >
> > You will have received a lot of spam mail from the dev mailing list now
> > for the issue migration of only the first 20 issues and their comments,
> > so I have temporarily halted the migration until I can sort this.
> >
>
> Crisis averted; we were able to disable the notifications and the
> migration is chugging along smoothly and silently.
>
> Cheers,
> -Tristan
>
> > Cheers,
> > -Tristan
> >
> >
> > On Wed, 2020-12-30 at 18:40 +0900, Tristan Van Berkom wrote:
> > > Hi all,
> > >
> > > I am about to run the migration script which will migrate issue and
> > > merge request and such records from gitlab to github.
> > >
> > > This is expected to take over 12 hours to complete, and is not
> > > expected
> > > to generate immense spam after having tested this before and making
> > > adjustments to the migration script.
> > >
> > > That said, things can always go wrong, as this is a one-off it will
> > > be
> > > best to let the script complete.
> > >
> > > I will not be setting the gitlab repository into a read-only
> > > archived
> > > state until after the migration completes, as this is untested, so:
> > >
> > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > > !!! !!!
> > > !!! Any new issues, merge requests, or comments !!!
> > > !!! made to gitlab after this point may very well !!!
> > > !!! be lost. !!!
> > > !!! !!!
> > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > >
> > > Please refrain from updating the buildstream repo in gitlab during
> > > this
> > > time.
> > >
> > > This has been tested a few times already, but of course many things
> > > can
> > > go wrong, fingers crossed !
> > >
> > > Cheers,
> > > -Tristan
> > >
> > > On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> > > > Hi all,
> > > >
> > > > As we have known for some time now, the BuildStream project will
> > > > be
> > > > moving to GitHub.
> > > >
> > > > Since we are in the holiday season and we are confident that it
> > > > will
> > > > not be disruptive, we are going to do our best to push this over
> > > > the
> > > > line during this lull, and kick off 2021 development using a
> > > > github
> > > > workflow.
> > > >
> > > > Migration plan
> > > > --------------
> > > > Chandan made a lot of progress some months ago to get CI mostly
> > > > working
> > > > in github[0].
> > > >
> > > > Tom Pollard, Douglas Winship, Chris Polin and I took another
> > > > crack
> > > > at
> > > > it last month and we've ironed out most of the creases to our
> > > > satisfaction.
> > > >
> > > > This mostly includes solving the release process and the problem
> > > > of automatically publishing documentation of prior releases.
> > > >
> > > > We created a writeup of our plan to perform the migration
> > > > here[1].
> > > >
> > > >
> > > > What are we migrating ?
> > > > -----------------------
> > > > We plan to migrate the following repositories to the apache
> > > > namespace
> > > > in github, at https://github.com/apache/.
> > > >
> > > > * The main BuildStream core repository
> > > > https://gitlab.com/BuildStream/buildstream/
> > > >
> > > > * The repository which generates the website
> > > > https://gitlab.com/BuildStream/website/
> > > >
> > > > * The repository which aggregates and publishes docs for released
> > > > versions of BuildStream
> > > > https://gitlab.com/BuildStream/docs-website/
> > > >
> > > > There are no plans to migrate the plugin repositories at this
> > > > time.
> > > >
> > > > While we may end up creating some new plugin specific
> > > > repositories
> > > > on
> > > > github in the future, we expect mainly to continue to use gitlab
> > > > for
> > > > plugin repositories in the long term, this will be easier for
> > > > plugin
> > > > maintainers to release asynchronously from core BuildStream and
> > > > with a
> > > > bit less bureaucracy.
> > > >
> > > >
> > > > BuildStream first
> > > > -----------------
> > > > We have recently acquired https://github.com/apache/buildstream/
> > > > for
> > > > our purposes to migrate the core BuildStream repository.
> > > >
> > > > All three repositories will not be migrated in one go,
> > > > BuildStream
> > > > will
> > > > be migrated first.
> > > >
> > > >
> > > > Loss
> > > > ----
> > > > Things may be a bit shaky for a time, there will be some minor
> > > > loss
> > > > in
> > > > the migration, but we've tested the migration and we're confident
> > > > that
> > > > loss is minimal.
> > > >
> > > > For example, some things like already merged/closed merge
> > > > requests
> > > > may
> > > > not be migrated as pull requests, and comments and issues will
> > > > not
> > > > automatically be assigned to your corresponding username on
> > > > github.
> > > >
> > > > Thanks to Douglas's scripting we have managed to preserve
> > > > references to
> > > > gitlab usernames referred to in comments: links to users in
> > > > historical
> > > > issues and merge requests will refer to users on gitlab, and not
> > > > users
> > > > which might not exist on github, or might exist but not be the
> > > > same
> > > > person.
> > > >
> > > > Some of the CI will have to be revived post migration, mostly
> > > > this
> > > > is
> > > > limited to remote execution testing - we have agreed to address
> > > > this
> > > > after the migration.
> > > >
> > > >
> > > > Summary
> > > > -------
> > > > With any luck, we will get this done swiftly by the time people
> > > > start
> > > > coming back to work in 2021, and I hope to see you all next year
> > > > on
> > > > github :)
> > > >
> > > > I will be periodically replying to this thread to announce
> > > > timeframes
> > > > as I coordinate with Apache folks and sort out permissions and
> > > > such.
> > > >
> > > > And of course, Merry Christmas and Happy New Year !!
> > > >
> > > > Cheers,
> > > > -Tristan
> > > >
> > > >
> > > > [0]:
> > > > https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> > > > [1]:
> > > > https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
> > > >
> > > >
> > > >
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi again,
On Wed, 2020-12-30 at 19:34 +0900, Tristan Van Berkom wrote:
> Hi all,
>
> As expected, something always goes wrong.
>
> You will have received a lot of spam mail from the dev mailing list now
> for the issue migration of only the first 20 issues and their comments,
> so I have temporarily halted the migration until I can sort this.
>
Crisis averted; we were able to disable the notifications and the
migration is chugging along smoothly and silently.
Cheers,
-Tristan
> Cheers,
> -Tristan
>
>
> On Wed, 2020-12-30 at 18:40 +0900, Tristan Van Berkom wrote:
> > Hi all,
> >
> > I am about to run the migration script which will migrate issue and
> > merge request and such records from gitlab to github.
> >
> > This is expected to take over 12 hours to complete, and is not
> > expected
> > to generate immense spam after having tested this before and making
> > adjustments to the migration script.
> >
> > That said, things can always go wrong, as this is a one-off it will
> > be
> > best to let the script complete.
> >
> > I will not be setting the gitlab repository into a read-only
> > archived
> > state until after the migration completes, as this is untested, so:
> >
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> > !!! !!!
> > !!! Any new issues, merge requests, or comments !!!
> > !!! made to gitlab after this point may very well !!!
> > !!! be lost. !!!
> > !!! !!!
> > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> >
> > Please refrain from updating the buildstream repo in gitlab during
> > this
> > time.
> >
> > This has been tested a few times already, but of course many things
> > can
> > go wrong, fingers crossed !
> >
> > Cheers,
> > -Tristan
> >
> > On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> > > Hi all,
> > >
> > > As we have known for some time now, the BuildStream project will
> > > be
> > > moving to GitHub.
> > >
> > > Since we are in the holiday season and we are confident that it
> > > will
> > > not be disruptive, we are going to do our best to push this over
> > > the
> > > line during this lull, and kick off 2021 development using a
> > > github
> > > workflow.
> > >
> > > Migration plan
> > > --------------
> > > Chandan made a lot of progress some months ago to get CI mostly
> > > working
> > > in github[0].
> > >
> > > Tom Pollard, Douglas Winship, Chris Polin and I took another
> > > crack
> > > at
> > > it last month and we've ironed out most of the creases to our
> > > satisfaction.
> > >
> > > This mostly includes solving the release process and the problem
> > > of automatically publishing documentation of prior releases.
> > >
> > > We created a writeup of our plan to perform the migration
> > > here[1].
> > >
> > >
> > > What are we migrating ?
> > > -----------------------
> > > We plan to migrate the following repositories to the apache
> > > namespace
> > > in github, at https://github.com/apache/.
> > >
> > > * The main BuildStream core repository
> > > https://gitlab.com/BuildStream/buildstream/
> > >
> > > * The repository which generates the website
> > > https://gitlab.com/BuildStream/website/
> > >
> > > * The repository which aggregates and publishes docs for released
> > > versions of BuildStream
> > > https://gitlab.com/BuildStream/docs-website/
> > >
> > > There are no plans to migrate the plugin repositories at this
> > > time.
> > >
> > > While we may end up creating some new plugin specific
> > > repositories
> > > on
> > > github in the future, we expect mainly to continue to use gitlab
> > > for
> > > plugin repositories in the long term, this will be easier for
> > > plugin
> > > maintainers to release asynchronously from core BuildStream and
> > > with a
> > > bit less bureaucracy.
> > >
> > >
> > > BuildStream first
> > > -----------------
> > > We have recently acquired https://github.com/apache/buildstream/
> > > for
> > > our purposes to migrate the core BuildStream repository.
> > >
> > > All three repositories will not be migrated in one go,
> > > BuildStream
> > > will
> > > be migrated first.
> > >
> > >
> > > Loss
> > > ----
> > > Things may be a bit shaky for a time, there will be some minor
> > > loss
> > > in
> > > the migration, but we've tested the migration and we're confident
> > > that
> > > loss is minimal.
> > >
> > > For example, some things like already merged/closed merge
> > > requests
> > > may
> > > not be migrated as pull requests, and comments and issues will
> > > not
> > > automatically be assigned to your corresponding username on
> > > github.
> > >
> > > Thanks to Douglas's scripting we have managed to preserve
> > > references to
> > > gitlab usernames referred to in comments: links to users in
> > > historical
> > > issues and merge requests will refer to users on gitlab, and not
> > > users
> > > which might not exist on github, or might exist but not be the
> > > same
> > > person.
> > >
> > > Some of the CI will have to be revived post migration, mostly
> > > this
> > > is
> > > limited to remote execution testing - we have agreed to address
> > > this
> > > after the migration.
> > >
> > >
> > > Summary
> > > -------
> > > With any luck, we will get this done swiftly by the time people
> > > start
> > > coming back to work in 2021, and I hope to see you all next year
> > > on
> > > github :)
> > >
> > > I will be periodically replying to this thread to announce
> > > timeframes
> > > as I coordinate with Apache folks and sort out permissions and
> > > such.
> > >
> > > And of course, Merry Christmas and Happy New Year !!
> > >
> > > Cheers,
> > > -Tristan
> > >
> > >
> > > [0]:
> > > https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> > > [1]:
> > > https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
> > >
> > >
> > >
>
>
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
As expected, something always goes wrong.
You will have received a lot of spam mail from the dev mailing list now
for the issue migration of only the first 20 issues and their comments,
so I have temporarily halted the migration until I can sort this.
Cheers,
-Tristan
On Wed, 2020-12-30 at 18:40 +0900, Tristan Van Berkom wrote:
> Hi all,
>
> I am about to run the migration script which will migrate issue and
> merge request and such records from gitlab to github.
>
> This is expected to take over 12 hours to complete, and is not
> expected
> to generate immense spam after having tested this before and making
> adjustments to the migration script.
>
> That said, things can always go wrong, as this is a one-off it will
> be
> best to let the script complete.
>
> I will not be setting the gitlab repository into a read-only archived
> state until after the migration completes, as this is untested, so:
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!! !!!
> !!! Any new issues, merge requests, or comments !!!
> !!! made to gitlab after this point may very well !!!
> !!! be lost. !!!
> !!! !!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
> Please refrain from updating the buildstream repo in gitlab during
> this
> time.
>
> This has been tested a few times already, but of course many things
> can
> go wrong, fingers crossed !
>
> Cheers,
> -Tristan
>
> On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> > Hi all,
> >
> > As we have known for some time now, the BuildStream project will be
> > moving to GitHub.
> >
> > Since we are in the holiday season and we are confident that it
> > will
> > not be disruptive, we are going to do our best to push this over
> > the
> > line during this lull, and kick off 2021 development using a github
> > workflow.
> >
> > Migration plan
> > --------------
> > Chandan made a lot of progress some months ago to get CI mostly
> > working
> > in github[0].
> >
> > Tom Pollard, Douglas Winship, Chris Polin and I took another crack
> > at
> > it last month and we've ironed out most of the creases to our
> > satisfaction.
> >
> > This mostly includes solving the release process and the problem
> > of automatically publishing documentation of prior releases.
> >
> > We created a writeup of our plan to perform the migration here[1].
> >
> >
> > What are we migrating ?
> > -----------------------
> > We plan to migrate the following repositories to the apache
> > namespace
> > in github, at https://github.com/apache/.
> >
> > * The main BuildStream core repository
> > https://gitlab.com/BuildStream/buildstream/
> >
> > * The repository which generates the website
> > https://gitlab.com/BuildStream/website/
> >
> > * The repository which aggregates and publishes docs for released
> > versions of BuildStream
> > https://gitlab.com/BuildStream/docs-website/
> >
> > There are no plans to migrate the plugin repositories at this time.
> >
> > While we may end up creating some new plugin specific repositories
> > on
> > github in the future, we expect mainly to continue to use gitlab
> > for
> > plugin repositories in the long term, this will be easier for
> > plugin
> > maintainers to release asynchronously from core BuildStream and
> > with a
> > bit less bureaucracy.
> >
> >
> > BuildStream first
> > -----------------
> > We have recently acquired https://github.com/apache/buildstream/
> > for
> > our purposes to migrate the core BuildStream repository.
> >
> > All three repositories will not be migrated in one go, BuildStream
> > will
> > be migrated first.
> >
> >
> > Loss
> > ----
> > Things may be a bit shaky for a time, there will be some minor loss
> > in
> > the migration, but we've tested the migration and we're confident
> > that
> > loss is minimal.
> >
> > For example, some things like already merged/closed merge requests
> > may
> > not be migrated as pull requests, and comments and issues will not
> > automatically be assigned to your corresponding username on github.
> >
> > Thanks to Douglas's scripting we have managed to preserve
> > references to
> > gitlab usernames referred to in comments: links to users in
> > historical
> > issues and merge requests will refer to users on gitlab, and not
> > users
> > which might not exist on github, or might exist but not be the same
> > person.
> >
> > Some of the CI will have to be revived post migration, mostly this
> > is
> > limited to remote execution testing - we have agreed to address
> > this
> > after the migration.
> >
> >
> > Summary
> > -------
> > With any luck, we will get this done swiftly by the time people
> > start
> > coming back to work in 2021, and I hope to see you all next year on
> > github :)
> >
> > I will be periodically replying to this thread to announce
> > timeframes
> > as I coordinate with Apache folks and sort out permissions and
> > such.
> >
> > And of course, Merry Christmas and Happy New Year !!
> >
> > Cheers,
> > -Tristan
> >
> >
> > [0]:
> > https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> > [1]:
> > https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
> >
> >
> >
>
>
Re: GitHub migration
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi all,
I am about to run the migration script which will migrate issue and
merge request and such records from gitlab to github.
This is expected to take over 12 hours to complete, and is not expected
to generate immense spam after having tested this before and making
adjustments to the migration script.
That said, things can always go wrong, as this is a one-off it will be
best to let the script complete.
I will not be setting the gitlab repository into a read-only archived
state until after the migration completes, as this is untested, so:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!! !!!
!!! Any new issues, merge requests, or comments !!!
!!! made to gitlab after this point may very well !!!
!!! be lost. !!!
!!! !!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Please refrain from updating the buildstream repo in gitlab during this
time.
This has been tested a few times already, but of course many things can
go wrong, fingers crossed !
Cheers,
-Tristan
On Mon, 2020-12-28 at 19:21 +0900, Tristan Van Berkom wrote:
> Hi all,
>
> As we have known for some time now, the BuildStream project will be
> moving to GitHub.
>
> Since we are in the holiday season and we are confident that it will
> not be disruptive, we are going to do our best to push this over the
> line during this lull, and kick off 2021 development using a github
> workflow.
>
> Migration plan
> --------------
> Chandan made a lot of progress some months ago to get CI mostly working
> in github[0].
>
> Tom Pollard, Douglas Winship, Chris Polin and I took another crack at
> it last month and we've ironed out most of the creases to our
> satisfaction.
>
> This mostly includes solving the release process and the problem
> of automatically publishing documentation of prior releases.
>
> We created a writeup of our plan to perform the migration here[1].
>
>
> What are we migrating ?
> -----------------------
> We plan to migrate the following repositories to the apache namespace
> in github, at https://github.com/apache/.
>
> * The main BuildStream core repository
> https://gitlab.com/BuildStream/buildstream/
>
> * The repository which generates the website
> https://gitlab.com/BuildStream/website/
>
> * The repository which aggregates and publishes docs for released
> versions of BuildStream
> https://gitlab.com/BuildStream/docs-website/
>
> There are no plans to migrate the plugin repositories at this time.
>
> While we may end up creating some new plugin specific repositories on
> github in the future, we expect mainly to continue to use gitlab for
> plugin repositories in the long term, this will be easier for plugin
> maintainers to release asynchronously from core BuildStream and with a
> bit less bureaucracy.
>
>
> BuildStream first
> -----------------
> We have recently acquired https://github.com/apache/buildstream/ for
> our purposes to migrate the core BuildStream repository.
>
> All three repositories will not be migrated in one go, BuildStream will
> be migrated first.
>
>
> Loss
> ----
> Things may be a bit shaky for a time, there will be some minor loss in
> the migration, but we've tested the migration and we're confident that
> loss is minimal.
>
> For example, some things like already merged/closed merge requests may
> not be migrated as pull requests, and comments and issues will not
> automatically be assigned to your corresponding username on github.
>
> Thanks to Douglas's scripting we have managed to preserve references to
> gitlab usernames referred to in comments: links to users in historical
> issues and merge requests will refer to users on gitlab, and not users
> which might not exist on github, or might exist but not be the same
> person.
>
> Some of the CI will have to be revived post migration, mostly this is
> limited to remote execution testing - we have agreed to address this
> after the migration.
>
>
> Summary
> -------
> With any luck, we will get this done swiftly by the time people start
> coming back to work in 2021, and I hope to see you all next year on
> github :)
>
> I will be periodically replying to this thread to announce timeframes
> as I coordinate with Apache folks and sort out permissions and such.
>
> And of course, Merry Christmas and Happy New Year !!
>
> Cheers,
> -Tristan
>
>
> [0]: https://lists.apache.org/thread.html/r229ff54d163389b3e8e42688174336117a139fad9fd63d034db1cc23%40%3Cdev.buildstream.apache.org%3E
> [1]: https://gitlab.com/BuildStream/buildstream/-/wikis/GitHubMigration
>
>
>
Re: GitHub migration
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Mon, Dec 28, 2020 at 5:22 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Hi all,
>
> As we have known for some time now, the BuildStream project will be
> moving to GitHub.
>
> Since we are in the holiday season and we are confident that it will
> not be disruptive, we are going to do our best to push this over the
> line during this lull, and kick off 2021 development using a github
> workflow.
>
Awesome - I should largely be around this week if anything comes up that I
might be able to help out with. I'm in the ASF Slack on #buildstream.
(I'm not on GNOME IRC, but I could hop on there if desired.)
On the GitHub Actions front, ASF infra just restricted the source of the GH
actions for our org. As long as the action sources are from an apache org
repository, we should be fine. Infra is whitelisting a few other orgs on
an vetted basis. https://issues.apache.org/jira/browse/INFRA-21234 has
more.
Cheers. -- justin