You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Wes McKinney <we...@gmail.com> on 2020/02/03 16:24:28 UTC
[DISCUSS] Improving GitHub Actions configurations + tooling
hi folks,
I have noticed that many of our GitHub Actions configurations are very
similar to each other
https://www.diffchecker.com/eF4tHdzo
Aside from the "copy-paste" issue, some work would have to be done to
generate a Crossbow configuration using GHA.
It seems like a solution to these issues is to create a program to
generate the GHA configurations (using some templates or other tools).
So what is in .github/workflows would not be edited by human hands in
general but rather generated by this program.
This program could also assist with local automation for
reproducibility purposes (for example, locally executing a cascade of
dependent docker-compose steps).
Thoughts?
- Wes
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Wes McKinney <we...@gmail.com>.
hi Kou,
Thanks for your thoughts.
My preference would be to generate _some_ configurations but not
necessarily _all_. The more copy-and-pasting is going on, the harder
it will be to refactor and make general improvements to some
configurations (since you may have to change N things instead of 1
thing). Since I haven't really worked much on the GitHub Actions files
I will leave the decisions to the people doing the work. If the builds
run and we can verify patches, then that's the important thing, but we
should monitor how much time is being spent manually editing and
refactoring these files over the course of this year.
- Wes
On Tue, Feb 4, 2020 at 8:43 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> Hi,
>
> -0 for this idea.
> I don't strongly against this idea but I don't like this
> idea.
>
>
> The current raw GitHub Actions configurations + raw Docker
> tools (+ raw shell scripts) are simple and easy to
> understand for many developers because there are no magics.
> If we create our tools, they will introduce some magics. It
> may block new developers. (We want to increase Apache Arrow
> developers as much as possible, right?)
>
>
> If we use raw GitHub Actions configuration, we can introduce
> knowledge from other projects. For example, I'll migrate
> MinGW builds on AppVeyor to GitHub Actions. I know how to
> setup MinGW environment on GitHub Actions:
>
> https://github.com/groonga/groonga/blob/master/.github/workflows/windows-mingw.yml
>
> If I can use raw GitHub Actions configuration, I'll re-use
> the similar approach in Apache Arrow easily. If we have our
> tools, I need to translate my GitHub Actions configuration
> knowledge to codes/configurations for our tools.
>
>
> I agree with duplication. But it may be resolved by
> improving GitHub Actions configuration syntax. For example,
> GitLab CI configuration has "include" feature:
>
> https://docs.gitlab.com/ee/ci/yaml/#include
>
> If GitHub Actions configuration implements similar feature,
> duplication will be able to reduced. GitHub accepts feature
> requests in community forum:
>
> https://github.community/t5/GitHub-Actions/bd-p/actions
>
> Feature request example:
>
> https://github.community/t5/GitHub-Actions/Feature-Request-context-property-similar-to-github-repository/m-p/38964/highlight/true#M3608
>
> No "include" like feature in GitHub Actions:
>
> https://github.community/t5/GitHub-Actions/Is-it-possible-to-reuse-workflow-yaml-to-setup-similar-workflows/m-p/40634#
>
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
> "Re: [DISCUSS] Improving GitHub Actions configurations + tooling" on Tue, 4 Feb 2020 22:54:14 +0100,
> Krisztián Szűcs <sz...@gmail.com> wrote:
>
> > On Tue, Feb 4, 2020 at 6:31 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >> I'm personally not too concerned with the details as long as people
> >> generally agree that the solution is maintainable (Kou's and Antoine's
> >> feedback here would be helpful) and there is not an abundant odor of
> >> code duplication
> > I'm not entirely satisfied with the current solution, but so far so good.
> >
> > Before improving the current GHA setup I'd be indeed nice to see others'
> > opinions and preferences.
> >>
> >> On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson
> >> <ne...@gmail.com> wrote:
> >> >
> >> > What if we wrote our own action(s) to wrap up some of the boilerplate? It
> >> > doesn't seem that there are any off-the-shelf actions we could use to drive
> >> > docker-compose:
> >> > https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose
> >> > but I don't think it would be that difficult to wrap `docker-compose pull
> >> > $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in
> >> > an action.
> >> >
> >> > Neal
> >> >
> >> > On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <sz...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
> >> > > >
> >> > > > hi folks,
> >> > > >
> >> > > > I have noticed that many of our GitHub Actions configurations are very
> >> > > > similar to each other
> >> > > They are indeed.
> >> > > >
> >> > > > https://www.diffchecker.com/eF4tHdzo
> >> > > >
> >> > > > Aside from the "copy-paste" issue, some work would have to be done to
> >> > > > generate a Crossbow configuration using GHA.
> >> > > Do you mean having GHA as a backend for crossbow? @Kou has just
> >> > > added support for that in [1]
> >> > > >
> >> > > > It seems like a solution to these issues is to create a program to
> >> > > > generate the GHA configurations (using some templates or other tools).
> >> > > > So what is in .github/workflows would not be edited by human hands in
> >> > > > general but rather generated by this program.
> >> > > That would be quite similar to what I've implemented in ursabot [2], just
> >> > > generating GHA flavored ymls instead of buildbot objects, so it seems
> >> > > doable. Of course we'll need commit hooks to force the regeneration of
> >> > > these configuration files.
> >> > > >
> >> > > > This program could also assist with local automation for
> >> > > > reproducibility purposes (for example, locally executing a cascade of
> >> > > > dependent docker-compose steps).
> >> > > Another independent improvement could be to ditch docker-compose
> >> > > completely.
> >> > > I'd say that 70% of the docker-compose.yml [3] and the relating
> >> > > dockerfiles are
> >> > > filled with duplications necessary because of the limited parametrization
> >> > > and
> >> > > reusability of docker and docker-compose. It also makes harder to use new
> >> > > docker
> >> > > features like https://docs.docker.com/buildx/working-with-buildx/
> >> > >
> >> > > Again I'm referring ursabot where I've already implemented the ideas, the
> >> > > docker files [4] and the image hierarchy from the compose file [3] could be
> >> > > replaced by something similar like the ursabot docker utility [6].
> >> > > The builder definitions [7] which are the counterparts for the scripts
> >> > > [8] compose
> >> > > command [9] and compose parameters [10].
> >> > > I'm not saying that we should use the exact build definition syntax
> >> > > from ursabot,
> >> > > we can have arbitrary ymls etc. but the configuration I've referenced from
> >> > > ursabot approximates what we can achieve by increasing the abstraction
> >> > > level.
> >> > >
> >> > > Note, that the ursabot sources I've referenced are not GPL contaminated.
> >> > > >
> >> > > > Thoughts?
> >> > > On the other hand I see one big advantage of the current setup. By using
> >> > > plain
> >> > > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
> >> > > scripts, our CI configuration is more idiomatic. It sits closer to the
> >> > > developers'
> >> > > expectations.
> >> > > It's hard for me to judge the developer friendliness of either of "CI
> >> > > tools"
> >> > > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
> >> > > of those, but I suppose that the new GHA setup is the easiest to work with.
> >> > > >
> >> > > Thanks, Krisztian
> >> > >
> >> > > [1] https://github.com/apache/arrow/pull/6286
> >> > > [2]
> >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
> >> > > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml
> >> > > [4] https://github.com/apache/arrow/tree/master/ci/docker
> >> > > [6]
> >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
> >> > > [7]
> >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
> >> > > [8] https://github.com/apache/arrow/tree/master/ci/scripts
> >> > > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
> >> > > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
> >> > > > - Wes
> >> > >
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
-0 for this idea.
I don't strongly against this idea but I don't like this
idea.
The current raw GitHub Actions configurations + raw Docker
tools (+ raw shell scripts) are simple and easy to
understand for many developers because there are no magics.
If we create our tools, they will introduce some magics. It
may block new developers. (We want to increase Apache Arrow
developers as much as possible, right?)
If we use raw GitHub Actions configuration, we can introduce
knowledge from other projects. For example, I'll migrate
MinGW builds on AppVeyor to GitHub Actions. I know how to
setup MinGW environment on GitHub Actions:
https://github.com/groonga/groonga/blob/master/.github/workflows/windows-mingw.yml
If I can use raw GitHub Actions configuration, I'll re-use
the similar approach in Apache Arrow easily. If we have our
tools, I need to translate my GitHub Actions configuration
knowledge to codes/configurations for our tools.
I agree with duplication. But it may be resolved by
improving GitHub Actions configuration syntax. For example,
GitLab CI configuration has "include" feature:
https://docs.gitlab.com/ee/ci/yaml/#include
If GitHub Actions configuration implements similar feature,
duplication will be able to reduced. GitHub accepts feature
requests in community forum:
https://github.community/t5/GitHub-Actions/bd-p/actions
Feature request example:
https://github.community/t5/GitHub-Actions/Feature-Request-context-property-similar-to-github-repository/m-p/38964/highlight/true#M3608
No "include" like feature in GitHub Actions:
https://github.community/t5/GitHub-Actions/Is-it-possible-to-reuse-workflow-yaml-to-setup-similar-workflows/m-p/40634#
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Re: [DISCUSS] Improving GitHub Actions configurations + tooling" on Tue, 4 Feb 2020 22:54:14 +0100,
Krisztián Szűcs <sz...@gmail.com> wrote:
> On Tue, Feb 4, 2020 at 6:31 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> I'm personally not too concerned with the details as long as people
>> generally agree that the solution is maintainable (Kou's and Antoine's
>> feedback here would be helpful) and there is not an abundant odor of
>> code duplication
> I'm not entirely satisfied with the current solution, but so far so good.
>
> Before improving the current GHA setup I'd be indeed nice to see others'
> opinions and preferences.
>>
>> On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson
>> <ne...@gmail.com> wrote:
>> >
>> > What if we wrote our own action(s) to wrap up some of the boilerplate? It
>> > doesn't seem that there are any off-the-shelf actions we could use to drive
>> > docker-compose:
>> > https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose
>> > but I don't think it would be that difficult to wrap `docker-compose pull
>> > $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in
>> > an action.
>> >
>> > Neal
>> >
>> > On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <sz...@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
>> > > >
>> > > > hi folks,
>> > > >
>> > > > I have noticed that many of our GitHub Actions configurations are very
>> > > > similar to each other
>> > > They are indeed.
>> > > >
>> > > > https://www.diffchecker.com/eF4tHdzo
>> > > >
>> > > > Aside from the "copy-paste" issue, some work would have to be done to
>> > > > generate a Crossbow configuration using GHA.
>> > > Do you mean having GHA as a backend for crossbow? @Kou has just
>> > > added support for that in [1]
>> > > >
>> > > > It seems like a solution to these issues is to create a program to
>> > > > generate the GHA configurations (using some templates or other tools).
>> > > > So what is in .github/workflows would not be edited by human hands in
>> > > > general but rather generated by this program.
>> > > That would be quite similar to what I've implemented in ursabot [2], just
>> > > generating GHA flavored ymls instead of buildbot objects, so it seems
>> > > doable. Of course we'll need commit hooks to force the regeneration of
>> > > these configuration files.
>> > > >
>> > > > This program could also assist with local automation for
>> > > > reproducibility purposes (for example, locally executing a cascade of
>> > > > dependent docker-compose steps).
>> > > Another independent improvement could be to ditch docker-compose
>> > > completely.
>> > > I'd say that 70% of the docker-compose.yml [3] and the relating
>> > > dockerfiles are
>> > > filled with duplications necessary because of the limited parametrization
>> > > and
>> > > reusability of docker and docker-compose. It also makes harder to use new
>> > > docker
>> > > features like https://docs.docker.com/buildx/working-with-buildx/
>> > >
>> > > Again I'm referring ursabot where I've already implemented the ideas, the
>> > > docker files [4] and the image hierarchy from the compose file [3] could be
>> > > replaced by something similar like the ursabot docker utility [6].
>> > > The builder definitions [7] which are the counterparts for the scripts
>> > > [8] compose
>> > > command [9] and compose parameters [10].
>> > > I'm not saying that we should use the exact build definition syntax
>> > > from ursabot,
>> > > we can have arbitrary ymls etc. but the configuration I've referenced from
>> > > ursabot approximates what we can achieve by increasing the abstraction
>> > > level.
>> > >
>> > > Note, that the ursabot sources I've referenced are not GPL contaminated.
>> > > >
>> > > > Thoughts?
>> > > On the other hand I see one big advantage of the current setup. By using
>> > > plain
>> > > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
>> > > scripts, our CI configuration is more idiomatic. It sits closer to the
>> > > developers'
>> > > expectations.
>> > > It's hard for me to judge the developer friendliness of either of "CI
>> > > tools"
>> > > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
>> > > of those, but I suppose that the new GHA setup is the easiest to work with.
>> > > >
>> > > Thanks, Krisztian
>> > >
>> > > [1] https://github.com/apache/arrow/pull/6286
>> > > [2]
>> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
>> > > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml
>> > > [4] https://github.com/apache/arrow/tree/master/ci/docker
>> > > [6]
>> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
>> > > [7]
>> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
>> > > [8] https://github.com/apache/arrow/tree/master/ci/scripts
>> > > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
>> > > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
>> > > > - Wes
>> > >
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Tue, Feb 4, 2020 at 6:31 PM Wes McKinney <we...@gmail.com> wrote:
>
> I'm personally not too concerned with the details as long as people
> generally agree that the solution is maintainable (Kou's and Antoine's
> feedback here would be helpful) and there is not an abundant odor of
> code duplication
I'm not entirely satisfied with the current solution, but so far so good.
Before improving the current GHA setup I'd be indeed nice to see others'
opinions and preferences.
>
> On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson
> <ne...@gmail.com> wrote:
> >
> > What if we wrote our own action(s) to wrap up some of the boilerplate? It
> > doesn't seem that there are any off-the-shelf actions we could use to drive
> > docker-compose:
> > https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose
> > but I don't think it would be that difficult to wrap `docker-compose pull
> > $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in
> > an action.
> >
> > Neal
> >
> > On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <sz...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > hi folks,
> > > >
> > > > I have noticed that many of our GitHub Actions configurations are very
> > > > similar to each other
> > > They are indeed.
> > > >
> > > > https://www.diffchecker.com/eF4tHdzo
> > > >
> > > > Aside from the "copy-paste" issue, some work would have to be done to
> > > > generate a Crossbow configuration using GHA.
> > > Do you mean having GHA as a backend for crossbow? @Kou has just
> > > added support for that in [1]
> > > >
> > > > It seems like a solution to these issues is to create a program to
> > > > generate the GHA configurations (using some templates or other tools).
> > > > So what is in .github/workflows would not be edited by human hands in
> > > > general but rather generated by this program.
> > > That would be quite similar to what I've implemented in ursabot [2], just
> > > generating GHA flavored ymls instead of buildbot objects, so it seems
> > > doable. Of course we'll need commit hooks to force the regeneration of
> > > these configuration files.
> > > >
> > > > This program could also assist with local automation for
> > > > reproducibility purposes (for example, locally executing a cascade of
> > > > dependent docker-compose steps).
> > > Another independent improvement could be to ditch docker-compose
> > > completely.
> > > I'd say that 70% of the docker-compose.yml [3] and the relating
> > > dockerfiles are
> > > filled with duplications necessary because of the limited parametrization
> > > and
> > > reusability of docker and docker-compose. It also makes harder to use new
> > > docker
> > > features like https://docs.docker.com/buildx/working-with-buildx/
> > >
> > > Again I'm referring ursabot where I've already implemented the ideas, the
> > > docker files [4] and the image hierarchy from the compose file [3] could be
> > > replaced by something similar like the ursabot docker utility [6].
> > > The builder definitions [7] which are the counterparts for the scripts
> > > [8] compose
> > > command [9] and compose parameters [10].
> > > I'm not saying that we should use the exact build definition syntax
> > > from ursabot,
> > > we can have arbitrary ymls etc. but the configuration I've referenced from
> > > ursabot approximates what we can achieve by increasing the abstraction
> > > level.
> > >
> > > Note, that the ursabot sources I've referenced are not GPL contaminated.
> > > >
> > > > Thoughts?
> > > On the other hand I see one big advantage of the current setup. By using
> > > plain
> > > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
> > > scripts, our CI configuration is more idiomatic. It sits closer to the
> > > developers'
> > > expectations.
> > > It's hard for me to judge the developer friendliness of either of "CI
> > > tools"
> > > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
> > > of those, but I suppose that the new GHA setup is the easiest to work with.
> > > >
> > > Thanks, Krisztian
> > >
> > > [1] https://github.com/apache/arrow/pull/6286
> > > [2]
> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
> > > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml
> > > [4] https://github.com/apache/arrow/tree/master/ci/docker
> > > [6]
> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
> > > [7]
> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
> > > [8] https://github.com/apache/arrow/tree/master/ci/scripts
> > > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
> > > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
> > > > - Wes
> > >
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Wes McKinney <we...@gmail.com>.
I'm personally not too concerned with the details as long as people
generally agree that the solution is maintainable (Kou's and Antoine's
feedback here would be helpful) and there is not an abundant odor of
code duplication
On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson
<ne...@gmail.com> wrote:
>
> What if we wrote our own action(s) to wrap up some of the boilerplate? It
> doesn't seem that there are any off-the-shelf actions we could use to drive
> docker-compose:
> https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose
> but I don't think it would be that difficult to wrap `docker-compose pull
> $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in
> an action.
>
> Neal
>
> On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <sz...@gmail.com>
> wrote:
>
> > Hi,
> >
> > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > hi folks,
> > >
> > > I have noticed that many of our GitHub Actions configurations are very
> > > similar to each other
> > They are indeed.
> > >
> > > https://www.diffchecker.com/eF4tHdzo
> > >
> > > Aside from the "copy-paste" issue, some work would have to be done to
> > > generate a Crossbow configuration using GHA.
> > Do you mean having GHA as a backend for crossbow? @Kou has just
> > added support for that in [1]
> > >
> > > It seems like a solution to these issues is to create a program to
> > > generate the GHA configurations (using some templates or other tools).
> > > So what is in .github/workflows would not be edited by human hands in
> > > general but rather generated by this program.
> > That would be quite similar to what I've implemented in ursabot [2], just
> > generating GHA flavored ymls instead of buildbot objects, so it seems
> > doable. Of course we'll need commit hooks to force the regeneration of
> > these configuration files.
> > >
> > > This program could also assist with local automation for
> > > reproducibility purposes (for example, locally executing a cascade of
> > > dependent docker-compose steps).
> > Another independent improvement could be to ditch docker-compose
> > completely.
> > I'd say that 70% of the docker-compose.yml [3] and the relating
> > dockerfiles are
> > filled with duplications necessary because of the limited parametrization
> > and
> > reusability of docker and docker-compose. It also makes harder to use new
> > docker
> > features like https://docs.docker.com/buildx/working-with-buildx/
> >
> > Again I'm referring ursabot where I've already implemented the ideas, the
> > docker files [4] and the image hierarchy from the compose file [3] could be
> > replaced by something similar like the ursabot docker utility [6].
> > The builder definitions [7] which are the counterparts for the scripts
> > [8] compose
> > command [9] and compose parameters [10].
> > I'm not saying that we should use the exact build definition syntax
> > from ursabot,
> > we can have arbitrary ymls etc. but the configuration I've referenced from
> > ursabot approximates what we can achieve by increasing the abstraction
> > level.
> >
> > Note, that the ursabot sources I've referenced are not GPL contaminated.
> > >
> > > Thoughts?
> > On the other hand I see one big advantage of the current setup. By using
> > plain
> > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
> > scripts, our CI configuration is more idiomatic. It sits closer to the
> > developers'
> > expectations.
> > It's hard for me to judge the developer friendliness of either of "CI
> > tools"
> > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
> > of those, but I suppose that the new GHA setup is the easiest to work with.
> > >
> > Thanks, Krisztian
> >
> > [1] https://github.com/apache/arrow/pull/6286
> > [2]
> > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
> > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml
> > [4] https://github.com/apache/arrow/tree/master/ci/docker
> > [6]
> > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
> > [7]
> > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
> > [8] https://github.com/apache/arrow/tree/master/ci/scripts
> > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
> > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
> > > - Wes
> >
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Neal Richardson <ne...@gmail.com>.
What if we wrote our own action(s) to wrap up some of the boilerplate? It
doesn't seem that there are any off-the-shelf actions we could use to drive
docker-compose:
https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose
but I don't think it would be that difficult to wrap `docker-compose pull
$JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in
an action.
Neal
On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <sz...@gmail.com>
wrote:
> Hi,
>
> On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi folks,
> >
> > I have noticed that many of our GitHub Actions configurations are very
> > similar to each other
> They are indeed.
> >
> > https://www.diffchecker.com/eF4tHdzo
> >
> > Aside from the "copy-paste" issue, some work would have to be done to
> > generate a Crossbow configuration using GHA.
> Do you mean having GHA as a backend for crossbow? @Kou has just
> added support for that in [1]
> >
> > It seems like a solution to these issues is to create a program to
> > generate the GHA configurations (using some templates or other tools).
> > So what is in .github/workflows would not be edited by human hands in
> > general but rather generated by this program.
> That would be quite similar to what I've implemented in ursabot [2], just
> generating GHA flavored ymls instead of buildbot objects, so it seems
> doable. Of course we'll need commit hooks to force the regeneration of
> these configuration files.
> >
> > This program could also assist with local automation for
> > reproducibility purposes (for example, locally executing a cascade of
> > dependent docker-compose steps).
> Another independent improvement could be to ditch docker-compose
> completely.
> I'd say that 70% of the docker-compose.yml [3] and the relating
> dockerfiles are
> filled with duplications necessary because of the limited parametrization
> and
> reusability of docker and docker-compose. It also makes harder to use new
> docker
> features like https://docs.docker.com/buildx/working-with-buildx/
>
> Again I'm referring ursabot where I've already implemented the ideas, the
> docker files [4] and the image hierarchy from the compose file [3] could be
> replaced by something similar like the ursabot docker utility [6].
> The builder definitions [7] which are the counterparts for the scripts
> [8] compose
> command [9] and compose parameters [10].
> I'm not saying that we should use the exact build definition syntax
> from ursabot,
> we can have arbitrary ymls etc. but the configuration I've referenced from
> ursabot approximates what we can achieve by increasing the abstraction
> level.
>
> Note, that the ursabot sources I've referenced are not GPL contaminated.
> >
> > Thoughts?
> On the other hand I see one big advantage of the current setup. By using
> plain
> GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
> scripts, our CI configuration is more idiomatic. It sits closer to the
> developers'
> expectations.
> It's hard for me to judge the developer friendliness of either of "CI
> tools"
> (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
> of those, but I suppose that the new GHA setup is the easiest to work with.
> >
> Thanks, Krisztian
>
> [1] https://github.com/apache/arrow/pull/6286
> [2]
> https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
> [3] https://github.com/apache/arrow/blob/master/docker-compose.yml
> [4] https://github.com/apache/arrow/tree/master/ci/docker
> [6]
> https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
> [7]
> https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
> [8] https://github.com/apache/arrow/tree/master/ci/scripts
> [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
> [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
> > - Wes
>
Re: [DISCUSS] Improving GitHub Actions configurations + tooling
Posted by Krisztián Szűcs <sz...@gmail.com>.
Hi,
On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi folks,
>
> I have noticed that many of our GitHub Actions configurations are very
> similar to each other
They are indeed.
>
> https://www.diffchecker.com/eF4tHdzo
>
> Aside from the "copy-paste" issue, some work would have to be done to
> generate a Crossbow configuration using GHA.
Do you mean having GHA as a backend for crossbow? @Kou has just
added support for that in [1]
>
> It seems like a solution to these issues is to create a program to
> generate the GHA configurations (using some templates or other tools).
> So what is in .github/workflows would not be edited by human hands in
> general but rather generated by this program.
That would be quite similar to what I've implemented in ursabot [2], just
generating GHA flavored ymls instead of buildbot objects, so it seems
doable. Of course we'll need commit hooks to force the regeneration of
these configuration files.
>
> This program could also assist with local automation for
> reproducibility purposes (for example, locally executing a cascade of
> dependent docker-compose steps).
Another independent improvement could be to ditch docker-compose completely.
I'd say that 70% of the docker-compose.yml [3] and the relating dockerfiles are
filled with duplications necessary because of the limited parametrization and
reusability of docker and docker-compose. It also makes harder to use new docker
features like https://docs.docker.com/buildx/working-with-buildx/
Again I'm referring ursabot where I've already implemented the ideas, the
docker files [4] and the image hierarchy from the compose file [3] could be
replaced by something similar like the ursabot docker utility [6].
The builder definitions [7] which are the counterparts for the scripts
[8] compose
command [9] and compose parameters [10].
I'm not saying that we should use the exact build definition syntax
from ursabot,
we can have arbitrary ymls etc. but the configuration I've referenced from
ursabot approximates what we can achieve by increasing the abstraction level.
Note, that the ursabot sources I've referenced are not GPL contaminated.
>
> Thoughts?
On the other hand I see one big advantage of the current setup. By using plain
GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash
scripts, our CI configuration is more idiomatic. It sits closer to the
developers'
expectations.
It's hard for me to judge the developer friendliness of either of "CI tools"
(GHA+docker setup, crossbow, ursabot) because I was the main perpetrator
of those, but I suppose that the new GHA setup is the easiest to work with.
>
Thanks, Krisztian
[1] https://github.com/apache/arrow/pull/6286
[2] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256
[3] https://github.com/apache/arrow/blob/master/docker-compose.yml
[4] https://github.com/apache/arrow/tree/master/ci/docker
[6] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py
[7] https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332
[8] https://github.com/apache/arrow/tree/master/ci/scripts
[9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306
[10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264
> - Wes