You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Philipp Zehnder <ze...@apache.org> on 2021/08/18 09:51:28 UTC
[Discuss] Optimize Github build
Hi all,
I have a question about our build in github, once we merged all the repos.
I recently tried to learn more about GitHub actions and was looking how we could improve our built script.
I came across this action here [1] to build and deploy docker images. Is that something that we could use?
Since we have many different containers it looks like it would would further simplify our workflow.
What do you think?
Philipp
[1] https://github.com/marketplace/actions/build-and-push-docker-images <https://github.com/marketplace/actions/build-and-push-docker-images>
> On 17. Aug 2021, at 21:54, Philipp Zehnder <ze...@apache.org> wrote:
>
> Hi Dominik,
>
> yes this is a good idea, it is a good chance to clean up our branches.
>
> Philipp
>
>> On 17. Aug 2021, at 21:51, Dominik Riemer <ri...@apache.org> wrote:
>>
>> Hi,
>>
>> to proceed with the integration of the extensions and installer repo into the main project, I'll soon start to clean up some branches that are no longer needed - it would be great if you could have a look and also remove merged or now obsolete branches.
>>
>> Dominik
>>
>> On 2021/07/27 11:01:27, Dominik Riemer <ri...@apache.org> wrote:
>>> Hi,
>>>
>>> sounds that there is a strong preference towards merging all three repos.
>>>
>>> So I guess there are at least two open questions:
>>> * @Chris Should we have a vote on this? I think that removing a source repo requires a vote as per ASF rules?
>>> * How do we technically merge the repos? It would be nice to preserve the Git history, but I'm not sure how we can handle this as I assume there are also a few branches that would need to be merged into individual branches of the core project.
>>>
>>> Any thoughts and experiences are welcome!
>>>
>>> Dominik
>>>
>>> On 2021/07/21 11:52:56, Patrick Wiener <wi...@apache.org> wrote:
>>>> +1 for merging everything into one code base including the installation options.
>>>>
>>>> It’s related anyways and would also make maintaining and developing things much easier
>>>>
>>>> Patrick
>>>>
>>>>> Am 21.07.2021 um 13:44 schrieb Dominik Riemer <ri...@apache.org>:
>>>>>
>>>>> Just a second-thought addition:
>>>>> Maybe we should indeed consider to also merge the whole installer into the core project - I think most users will survive downloading a larger bundle and we should probably try to make the release process as simple as possible so that we can have future releases more frequently and also don't require a 6-week release process.
>>>>>
>>>>> Dominik
>>>>>
>>>>> On 2021/07/21 11:32:22, Dominik Riemer <ri...@apache.org> wrote:
>>>>>> Hi Chris,
>>>>>>
>>>>>> thanks for bringing this up and a strong +1 for simplifying the release process.
>>>>>>
>>>>>> We already had this discussion in the past and didn't follow up on this.
>>>>>>
>>>>>> My preference would be to merge the incubator-streampipes-extensions project into the main project to have a single build for the whole project. Concerning the installer, I'd rather prefer to keep it as a standalone project as this should be just a minimal-size artifact to provide installation scripts for Docker, K8s and the CLI. But as the installer doesn't work anyways at release validation time as the artifacts are not yet available, it could be a separate vote process and we could also easily reduce the frequency of releases by pointing to the latest release in the version tag.
>>>>>>
>>>>>> What do others think? I'd personally like to at least merge the extensions and core projects soon.
>>>>>>
>>>>>> Dominik
>>>>>>
>>>>>>
>>>>>> On 2021/07/21 08:48:00, Christofer Dutz <ch...@c-ware.de> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have observed quite a bit of hesitation of IPMC votes to come in.
>>>>>>> One thing I noticed when reviewing your latest release (and the prior ones):
>>>>>>>
>>>>>>> You currently have 3 release artifacts:
>>>>>>> - Core
>>>>>>> - Extensions
>>>>>>> - Installer
>>>>>>>
>>>>>>> Unfortunately thoroughly doing a release validation is like doing 3
>>>>>>> release validations as most of the steps involved have to be executed 3
>>>>>>> times.
>>>>>>>
>>>>>>> This definitely caused a bit of frustration when I was finished with the
>>>>>>> first artifact and noticed there were 2 more to go. I could imagine this
>>>>>>> could be something that might be keeping the others from voting.
>>>>>>>
>>>>>>> Would it be possible to merge the release artifacts to one big bundle?
>>>>>>> In that case it would drastically reduce the amount of work.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>
Re: AW: [Discuss] Optimize Github build
Posted by Dominik Riemer <ri...@apache.org>.
Sorry, meant Airflow, not Arrow...
On 2021/08/18 14:39:04, Dominik Riemer <ri...@apache.org> wrote:
> Hi Philipp,
>
> as Chris mentioned, we can use Github Actions for everything that does not require ASF credentials.
> For instance, we can build, test and also push Docker images to our Docker Hub (Infra has already added our Dockerhub credentials to our Github account), but we can, e.g., not deploy to Maven repos (that's why we have a separate Jenkins job which publishes snapshots to the ASF Nexus).
>
> The action you mentioned looks very good, especially as it seems to support multi-arch Docker builds, so I think this can greatly simplify our current Github Actions build script.
>
> The Apache Arrow community has gathered a lot of experience with Github Actions and they published a very informative wiki document which also highlights potential security pitfalls related to Github Actions [1].
>
> Dominik
>
> [1] https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
>
> On 2021/08/18 09:58:46, Christofer Dutz <ch...@c-ware.de> wrote:
> > I think you should ask on builds@apache.org regarding this.
> > We can't do everything we do on Jenkins in GitHub actions as this would require technical user accounts with permissions to push to Apache infra outside of the control of Infra.
> >
> > So before investing too much time on this, I would suggest you ask first.
> >
> > Chris
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Philipp Zehnder <ze...@apache.org>
> > Gesendet: Mittwoch, 18. August 2021 11:51
> > An: dev@streampipes.apache.org
> > Betreff: [Discuss] Optimize Github build
> >
> > Hi all,
> >
> > I have a question about our build in github, once we merged all the repos.
> >
> > I recently tried to learn more about GitHub actions and was looking how we could improve our built script.
> > I came across this action here [1] to build and deploy docker images. Is that something that we could use?
> > Since we have many different containers it looks like it would would further simplify our workflow.
> >
> > What do you think?
> >
> > Philipp
> > [1] https://github.com/marketplace/actions/build-and-push-docker-images <https://github.com/marketplace/actions/build-and-push-docker-images>
> >
> >
> >
> > > On 17. Aug 2021, at 21:54, Philipp Zehnder <ze...@apache.org> wrote:
> > >
> > > Hi Dominik,
> > >
> > > yes this is a good idea, it is a good chance to clean up our branches.
> > >
> > > Philipp
> > >
> > >> On 17. Aug 2021, at 21:51, Dominik Riemer <ri...@apache.org> wrote:
> > >>
> > >> Hi,
> > >>
> > >> to proceed with the integration of the extensions and installer repo into the main project, I'll soon start to clean up some branches that are no longer needed - it would be great if you could have a look and also remove merged or now obsolete branches.
> > >>
> > >> Dominik
> > >>
> > >> On 2021/07/27 11:01:27, Dominik Riemer <ri...@apache.org> wrote:
> > >>> Hi,
> > >>>
> > >>> sounds that there is a strong preference towards merging all three repos.
> > >>>
> > >>> So I guess there are at least two open questions:
> > >>> * @Chris Should we have a vote on this? I think that removing a source repo requires a vote as per ASF rules?
> > >>> * How do we technically merge the repos? It would be nice to preserve the Git history, but I'm not sure how we can handle this as I assume there are also a few branches that would need to be merged into individual branches of the core project.
> > >>>
> > >>> Any thoughts and experiences are welcome!
> > >>>
> > >>> Dominik
> > >>>
> > >>> On 2021/07/21 11:52:56, Patrick Wiener <wi...@apache.org> wrote:
> > >>>> +1 for merging everything into one code base including the installation options.
> > >>>>
> > >>>> It’s related anyways and would also make maintaining and developing
> > >>>> things much easier
> > >>>>
> > >>>> Patrick
> > >>>>
> > >>>>> Am 21.07.2021 um 13:44 schrieb Dominik Riemer <ri...@apache.org>:
> > >>>>>
> > >>>>> Just a second-thought addition:
> > >>>>> Maybe we should indeed consider to also merge the whole installer into the core project - I think most users will survive downloading a larger bundle and we should probably try to make the release process as simple as possible so that we can have future releases more frequently and also don't require a 6-week release process.
> > >>>>>
> > >>>>> Dominik
> > >>>>>
> > >>>>> On 2021/07/21 11:32:22, Dominik Riemer <ri...@apache.org> wrote:
> > >>>>>> Hi Chris,
> > >>>>>>
> > >>>>>> thanks for bringing this up and a strong +1 for simplifying the release process.
> > >>>>>>
> > >>>>>> We already had this discussion in the past and didn't follow up on this.
> > >>>>>>
> > >>>>>> My preference would be to merge the incubator-streampipes-extensions project into the main project to have a single build for the whole project. Concerning the installer, I'd rather prefer to keep it as a standalone project as this should be just a minimal-size artifact to provide installation scripts for Docker, K8s and the CLI. But as the installer doesn't work anyways at release validation time as the artifacts are not yet available, it could be a separate vote process and we could also easily reduce the frequency of releases by pointing to the latest release in the version tag.
> > >>>>>>
> > >>>>>> What do others think? I'd personally like to at least merge the extensions and core projects soon.
> > >>>>>>
> > >>>>>> Dominik
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2021/07/21 08:48:00, Christofer Dutz <ch...@c-ware.de> wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> I have observed quite a bit of hesitation of IPMC votes to come in.
> > >>>>>>> One thing I noticed when reviewing your latest release (and the prior ones):
> > >>>>>>>
> > >>>>>>> You currently have 3 release artifacts:
> > >>>>>>> - Core
> > >>>>>>> - Extensions
> > >>>>>>> - Installer
> > >>>>>>>
> > >>>>>>> Unfortunately thoroughly doing a release validation is like
> > >>>>>>> doing 3 release validations as most of the steps involved have
> > >>>>>>> to be executed 3 times.
> > >>>>>>>
> > >>>>>>> This definitely caused a bit of frustration when I was finished
> > >>>>>>> with the first artifact and noticed there were 2 more to go. I
> > >>>>>>> could imagine this could be something that might be keeping the others from voting.
> > >>>>>>>
> > >>>>>>> Would it be possible to merge the release artifacts to one big bundle?
> > >>>>>>> In that case it would drastically reduce the amount of work.
> > >>>>>>>
> > >>>>>>> Chris
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >
> >
> >
>
Re: AW: [Discuss] Optimize Github build
Posted by Dominik Riemer <ri...@apache.org>.
Hi Philipp,
as Chris mentioned, we can use Github Actions for everything that does not require ASF credentials.
For instance, we can build, test and also push Docker images to our Docker Hub (Infra has already added our Dockerhub credentials to our Github account), but we can, e.g., not deploy to Maven repos (that's why we have a separate Jenkins job which publishes snapshots to the ASF Nexus).
The action you mentioned looks very good, especially as it seems to support multi-arch Docker builds, so I think this can greatly simplify our current Github Actions build script.
The Apache Arrow community has gathered a lot of experience with Github Actions and they published a very informative wiki document which also highlights potential security pitfalls related to Github Actions [1].
Dominik
[1] https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
On 2021/08/18 09:58:46, Christofer Dutz <ch...@c-ware.de> wrote:
> I think you should ask on builds@apache.org regarding this.
> We can't do everything we do on Jenkins in GitHub actions as this would require technical user accounts with permissions to push to Apache infra outside of the control of Infra.
>
> So before investing too much time on this, I would suggest you ask first.
>
> Chris
>
> -----Ursprüngliche Nachricht-----
> Von: Philipp Zehnder <ze...@apache.org>
> Gesendet: Mittwoch, 18. August 2021 11:51
> An: dev@streampipes.apache.org
> Betreff: [Discuss] Optimize Github build
>
> Hi all,
>
> I have a question about our build in github, once we merged all the repos.
>
> I recently tried to learn more about GitHub actions and was looking how we could improve our built script.
> I came across this action here [1] to build and deploy docker images. Is that something that we could use?
> Since we have many different containers it looks like it would would further simplify our workflow.
>
> What do you think?
>
> Philipp
> [1] https://github.com/marketplace/actions/build-and-push-docker-images <https://github.com/marketplace/actions/build-and-push-docker-images>
>
>
>
> > On 17. Aug 2021, at 21:54, Philipp Zehnder <ze...@apache.org> wrote:
> >
> > Hi Dominik,
> >
> > yes this is a good idea, it is a good chance to clean up our branches.
> >
> > Philipp
> >
> >> On 17. Aug 2021, at 21:51, Dominik Riemer <ri...@apache.org> wrote:
> >>
> >> Hi,
> >>
> >> to proceed with the integration of the extensions and installer repo into the main project, I'll soon start to clean up some branches that are no longer needed - it would be great if you could have a look and also remove merged or now obsolete branches.
> >>
> >> Dominik
> >>
> >> On 2021/07/27 11:01:27, Dominik Riemer <ri...@apache.org> wrote:
> >>> Hi,
> >>>
> >>> sounds that there is a strong preference towards merging all three repos.
> >>>
> >>> So I guess there are at least two open questions:
> >>> * @Chris Should we have a vote on this? I think that removing a source repo requires a vote as per ASF rules?
> >>> * How do we technically merge the repos? It would be nice to preserve the Git history, but I'm not sure how we can handle this as I assume there are also a few branches that would need to be merged into individual branches of the core project.
> >>>
> >>> Any thoughts and experiences are welcome!
> >>>
> >>> Dominik
> >>>
> >>> On 2021/07/21 11:52:56, Patrick Wiener <wi...@apache.org> wrote:
> >>>> +1 for merging everything into one code base including the installation options.
> >>>>
> >>>> It’s related anyways and would also make maintaining and developing
> >>>> things much easier
> >>>>
> >>>> Patrick
> >>>>
> >>>>> Am 21.07.2021 um 13:44 schrieb Dominik Riemer <ri...@apache.org>:
> >>>>>
> >>>>> Just a second-thought addition:
> >>>>> Maybe we should indeed consider to also merge the whole installer into the core project - I think most users will survive downloading a larger bundle and we should probably try to make the release process as simple as possible so that we can have future releases more frequently and also don't require a 6-week release process.
> >>>>>
> >>>>> Dominik
> >>>>>
> >>>>> On 2021/07/21 11:32:22, Dominik Riemer <ri...@apache.org> wrote:
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> thanks for bringing this up and a strong +1 for simplifying the release process.
> >>>>>>
> >>>>>> We already had this discussion in the past and didn't follow up on this.
> >>>>>>
> >>>>>> My preference would be to merge the incubator-streampipes-extensions project into the main project to have a single build for the whole project. Concerning the installer, I'd rather prefer to keep it as a standalone project as this should be just a minimal-size artifact to provide installation scripts for Docker, K8s and the CLI. But as the installer doesn't work anyways at release validation time as the artifacts are not yet available, it could be a separate vote process and we could also easily reduce the frequency of releases by pointing to the latest release in the version tag.
> >>>>>>
> >>>>>> What do others think? I'd personally like to at least merge the extensions and core projects soon.
> >>>>>>
> >>>>>> Dominik
> >>>>>>
> >>>>>>
> >>>>>> On 2021/07/21 08:48:00, Christofer Dutz <ch...@c-ware.de> wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I have observed quite a bit of hesitation of IPMC votes to come in.
> >>>>>>> One thing I noticed when reviewing your latest release (and the prior ones):
> >>>>>>>
> >>>>>>> You currently have 3 release artifacts:
> >>>>>>> - Core
> >>>>>>> - Extensions
> >>>>>>> - Installer
> >>>>>>>
> >>>>>>> Unfortunately thoroughly doing a release validation is like
> >>>>>>> doing 3 release validations as most of the steps involved have
> >>>>>>> to be executed 3 times.
> >>>>>>>
> >>>>>>> This definitely caused a bit of frustration when I was finished
> >>>>>>> with the first artifact and noticed there were 2 more to go. I
> >>>>>>> could imagine this could be something that might be keeping the others from voting.
> >>>>>>>
> >>>>>>> Would it be possible to merge the release artifacts to one big bundle?
> >>>>>>> In that case it would drastically reduce the amount of work.
> >>>>>>>
> >>>>>>> Chris
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >
>
>
AW: [Discuss] Optimize Github build
Posted by Christofer Dutz <ch...@c-ware.de>.
I think you should ask on builds@apache.org regarding this.
We can't do everything we do on Jenkins in GitHub actions as this would require technical user accounts with permissions to push to Apache infra outside of the control of Infra.
So before investing too much time on this, I would suggest you ask first.
Chris
-----Ursprüngliche Nachricht-----
Von: Philipp Zehnder <ze...@apache.org>
Gesendet: Mittwoch, 18. August 2021 11:51
An: dev@streampipes.apache.org
Betreff: [Discuss] Optimize Github build
Hi all,
I have a question about our build in github, once we merged all the repos.
I recently tried to learn more about GitHub actions and was looking how we could improve our built script.
I came across this action here [1] to build and deploy docker images. Is that something that we could use?
Since we have many different containers it looks like it would would further simplify our workflow.
What do you think?
Philipp
[1] https://github.com/marketplace/actions/build-and-push-docker-images <https://github.com/marketplace/actions/build-and-push-docker-images>
> On 17. Aug 2021, at 21:54, Philipp Zehnder <ze...@apache.org> wrote:
>
> Hi Dominik,
>
> yes this is a good idea, it is a good chance to clean up our branches.
>
> Philipp
>
>> On 17. Aug 2021, at 21:51, Dominik Riemer <ri...@apache.org> wrote:
>>
>> Hi,
>>
>> to proceed with the integration of the extensions and installer repo into the main project, I'll soon start to clean up some branches that are no longer needed - it would be great if you could have a look and also remove merged or now obsolete branches.
>>
>> Dominik
>>
>> On 2021/07/27 11:01:27, Dominik Riemer <ri...@apache.org> wrote:
>>> Hi,
>>>
>>> sounds that there is a strong preference towards merging all three repos.
>>>
>>> So I guess there are at least two open questions:
>>> * @Chris Should we have a vote on this? I think that removing a source repo requires a vote as per ASF rules?
>>> * How do we technically merge the repos? It would be nice to preserve the Git history, but I'm not sure how we can handle this as I assume there are also a few branches that would need to be merged into individual branches of the core project.
>>>
>>> Any thoughts and experiences are welcome!
>>>
>>> Dominik
>>>
>>> On 2021/07/21 11:52:56, Patrick Wiener <wi...@apache.org> wrote:
>>>> +1 for merging everything into one code base including the installation options.
>>>>
>>>> It’s related anyways and would also make maintaining and developing
>>>> things much easier
>>>>
>>>> Patrick
>>>>
>>>>> Am 21.07.2021 um 13:44 schrieb Dominik Riemer <ri...@apache.org>:
>>>>>
>>>>> Just a second-thought addition:
>>>>> Maybe we should indeed consider to also merge the whole installer into the core project - I think most users will survive downloading a larger bundle and we should probably try to make the release process as simple as possible so that we can have future releases more frequently and also don't require a 6-week release process.
>>>>>
>>>>> Dominik
>>>>>
>>>>> On 2021/07/21 11:32:22, Dominik Riemer <ri...@apache.org> wrote:
>>>>>> Hi Chris,
>>>>>>
>>>>>> thanks for bringing this up and a strong +1 for simplifying the release process.
>>>>>>
>>>>>> We already had this discussion in the past and didn't follow up on this.
>>>>>>
>>>>>> My preference would be to merge the incubator-streampipes-extensions project into the main project to have a single build for the whole project. Concerning the installer, I'd rather prefer to keep it as a standalone project as this should be just a minimal-size artifact to provide installation scripts for Docker, K8s and the CLI. But as the installer doesn't work anyways at release validation time as the artifacts are not yet available, it could be a separate vote process and we could also easily reduce the frequency of releases by pointing to the latest release in the version tag.
>>>>>>
>>>>>> What do others think? I'd personally like to at least merge the extensions and core projects soon.
>>>>>>
>>>>>> Dominik
>>>>>>
>>>>>>
>>>>>> On 2021/07/21 08:48:00, Christofer Dutz <ch...@c-ware.de> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have observed quite a bit of hesitation of IPMC votes to come in.
>>>>>>> One thing I noticed when reviewing your latest release (and the prior ones):
>>>>>>>
>>>>>>> You currently have 3 release artifacts:
>>>>>>> - Core
>>>>>>> - Extensions
>>>>>>> - Installer
>>>>>>>
>>>>>>> Unfortunately thoroughly doing a release validation is like
>>>>>>> doing 3 release validations as most of the steps involved have
>>>>>>> to be executed 3 times.
>>>>>>>
>>>>>>> This definitely caused a bit of frustration when I was finished
>>>>>>> with the first artifact and noticed there were 2 more to go. I
>>>>>>> could imagine this could be something that might be keeping the others from voting.
>>>>>>>
>>>>>>> Would it be possible to merge the release artifacts to one big bundle?
>>>>>>> In that case it would drastically reduce the amount of work.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>