You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Ziyao Wei <zi...@streamnative.io.INVALID> on 2022/03/19 04:32:24 UTC
[DISCUSS] Create a nightly docker image
Hi Pulsar developers,
Recently on Docker Hub the `latest` image has been changed from 2.8.2 to
2.9.1. As a result, the Golang client CI is failing. I have been debugging
the issue, but to help timely detection of similar issues in the future and
to ease debugging, I propose we should create a nightly Docker image for
Pulsar builds.
I filed https://github.com/apache/pulsar/issues/14755, and the arguments
are as follows:
*Is your enhancement request related to a problem? Please describe.*
Currently Docker Hub only contains release images for Pulsar. This makes
doing CI for dependent repos harder, since they wouldn't be able to test
again the bleeding edge easily, and has led to issues such as
https://github.com/apache/pulsar-client-go/issues/748 where it's unclear
which Pulsar change caused the issue.
(The linked issue was made worse because the latest image has been outdated
by ~6 months, but 1) even if that's not the case we'd still only be able to
catch issues between releases, and 2) the fact that the Golang client is
using latest points to a need of having a nightly.)
*Describe the solution you'd **like*
Create a nightly build so external repos can always point against it and
catch issues as early as possible.
*Describe alternatives you've considered*
The external repos can also try to pull in the latest Pulsar code, but that
seems harder and needs more per repo configuration.
@eolivelli suggested I post this here so it can be discussed. Opinions and
ideas are welcome, thanks!
Best,
Ziyao
Re: [DISCUSS] Create a nightly docker image
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno mar 22 mar 2022 alle ore 15:21 Ziyao Wei
<zi...@streamnative.io.invalid> ha scritto:
>
> Cool, thanks for the ideas! I agree it makes sense to separate the
> nightlies from the other versions.
>
> Just curious what's the advantage of using Artifactory over using Docker
> Hub? I read it has better access control but I don't have any experience
> with it.
The problem is that people MUST NOT think that those images can be
used in production.
If something is in DockerHub then it is easier for people to think
that it can be used in production.
>
> Also, I could send an email out to builds@apache.org to ask for access, but
> since I am not a committer/project member yet, would it be better if
> someone else closer to Apache does it?
I can help. You can DM me in Slack
>
>
> Best,
>
> Ziyao
>
>
> On Sat, Mar 19, 2022 at 7:13 AM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Il Sab 19 Mar 2022, 06:02 Dave Fisher <wa...@comcast.net> ha scritto:
> >
> > > The ASF has a JFrog artifactory instance that could help.
> >
> >
> > +1
> >
> > It must be clear that they are official releases and they MUST not be used
> > in Production
> >
> > We could also publish snapshots of Maven jars to repository.apache.org
> >
> > Enrico
> >
> > Ask on builds@apache.org!
> > >
> > > Regards,
> > > Dave
> > >
> > > Sent from my iPhone
> > >
> > > > On Mar 18, 2022, at 9:53 PM, PengHui Li <pe...@apache.org> wrote:
> > > >
> > > > I think we can use a separate org to maintain the nightly builded
> > > snapshot
> > > > images?
> > > > So that the external repo can pull the latest image from the new org.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > >
> > > >
> > > >
> > > >> On Sat, Mar 19, 2022 at 12:33 PM Ziyao Wei
> > > >> <zi...@streamnative.io.invalid> wrote:
> > > >>
> > > >> Hi Pulsar developers,
> > > >>
> > > >>
> > > >> Recently on Docker Hub the `latest` image has been changed from 2.8.2
> > to
> > > >> 2.9.1. As a result, the Golang client CI is failing. I have been
> > > debugging
> > > >> the issue, but to help timely detection of similar issues in the
> > future
> > > and
> > > >> to ease debugging, I propose we should create a nightly Docker image
> > for
> > > >> Pulsar builds.
> > > >>
> > > >> I filed https://github.com/apache/pulsar/issues/14755, and the
> > > arguments
> > > >> are as follows:
> > > >>
> > > >> *Is your enhancement request related to a problem? Please describe.*
> > > >>
> > > >> Currently Docker Hub only contains release images for Pulsar. This
> > makes
> > > >> doing CI for dependent repos harder, since they wouldn't be able to
> > test
> > > >> again the bleeding edge easily, and has led to issues such as
> > > >> https://github.com/apache/pulsar-client-go/issues/748 where it's
> > > unclear
> > > >> which Pulsar change caused the issue.
> > > >>
> > > >> (The linked issue was made worse because the latest image has been
> > > outdated
> > > >> by ~6 months, but 1) even if that's not the case we'd still only be
> > > able to
> > > >> catch issues between releases, and 2) the fact that the Golang client
> > is
> > > >> using latest points to a need of having a nightly.)
> > > >>
> > > >> *Describe the solution you'd **like*
> > > >>
> > > >> Create a nightly build so external repos can always point against it
> > and
> > > >> catch issues as early as possible.
> > > >>
> > > >> *Describe alternatives you've considered*
> > > >>
> > > >> The external repos can also try to pull in the latest Pulsar code, but
> > > that
> > > >> seems harder and needs more per repo configuration.
> > > >>
> > > >> @eolivelli suggested I post this here so it can be discussed. Opinions
> > > and
> > > >> ideas are welcome, thanks!
> > > >>
> > > >>
> > > >> Best,
> > > >>
> > > >> Ziyao
> > > >>
> > >
> > >
> >
Re: [DISCUSS] Create a nightly docker image
Posted by Ziyao Wei <zi...@streamnative.io.INVALID>.
Cool, thanks for the ideas! I agree it makes sense to separate the
nightlies from the other versions.
Just curious what's the advantage of using Artifactory over using Docker
Hub? I read it has better access control but I don't have any experience
with it.
Also, I could send an email out to builds@apache.org to ask for access, but
since I am not a committer/project member yet, would it be better if
someone else closer to Apache does it?
Best,
Ziyao
On Sat, Mar 19, 2022 at 7:13 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Il Sab 19 Mar 2022, 06:02 Dave Fisher <wa...@comcast.net> ha scritto:
>
> > The ASF has a JFrog artifactory instance that could help.
>
>
> +1
>
> It must be clear that they are official releases and they MUST not be used
> in Production
>
> We could also publish snapshots of Maven jars to repository.apache.org
>
> Enrico
>
> Ask on builds@apache.org!
> >
> > Regards,
> > Dave
> >
> > Sent from my iPhone
> >
> > > On Mar 18, 2022, at 9:53 PM, PengHui Li <pe...@apache.org> wrote:
> > >
> > > I think we can use a separate org to maintain the nightly builded
> > snapshot
> > > images?
> > > So that the external repo can pull the latest image from the new org.
> > >
> > > Regards,
> > > Penghui
> > >
> > >
> > >
> > >
> > >> On Sat, Mar 19, 2022 at 12:33 PM Ziyao Wei
> > >> <zi...@streamnative.io.invalid> wrote:
> > >>
> > >> Hi Pulsar developers,
> > >>
> > >>
> > >> Recently on Docker Hub the `latest` image has been changed from 2.8.2
> to
> > >> 2.9.1. As a result, the Golang client CI is failing. I have been
> > debugging
> > >> the issue, but to help timely detection of similar issues in the
> future
> > and
> > >> to ease debugging, I propose we should create a nightly Docker image
> for
> > >> Pulsar builds.
> > >>
> > >> I filed https://github.com/apache/pulsar/issues/14755, and the
> > arguments
> > >> are as follows:
> > >>
> > >> *Is your enhancement request related to a problem? Please describe.*
> > >>
> > >> Currently Docker Hub only contains release images for Pulsar. This
> makes
> > >> doing CI for dependent repos harder, since they wouldn't be able to
> test
> > >> again the bleeding edge easily, and has led to issues such as
> > >> https://github.com/apache/pulsar-client-go/issues/748 where it's
> > unclear
> > >> which Pulsar change caused the issue.
> > >>
> > >> (The linked issue was made worse because the latest image has been
> > outdated
> > >> by ~6 months, but 1) even if that's not the case we'd still only be
> > able to
> > >> catch issues between releases, and 2) the fact that the Golang client
> is
> > >> using latest points to a need of having a nightly.)
> > >>
> > >> *Describe the solution you'd **like*
> > >>
> > >> Create a nightly build so external repos can always point against it
> and
> > >> catch issues as early as possible.
> > >>
> > >> *Describe alternatives you've considered*
> > >>
> > >> The external repos can also try to pull in the latest Pulsar code, but
> > that
> > >> seems harder and needs more per repo configuration.
> > >>
> > >> @eolivelli suggested I post this here so it can be discussed. Opinions
> > and
> > >> ideas are welcome, thanks!
> > >>
> > >>
> > >> Best,
> > >>
> > >> Ziyao
> > >>
> >
> >
>
Re: [DISCUSS] Create a nightly docker image
Posted by Enrico Olivelli <eo...@gmail.com>.
Il Sab 19 Mar 2022, 06:02 Dave Fisher <wa...@comcast.net> ha scritto:
> The ASF has a JFrog artifactory instance that could help.
+1
It must be clear that they are official releases and they MUST not be used
in Production
We could also publish snapshots of Maven jars to repository.apache.org
Enrico
Ask on builds@apache.org!
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Mar 18, 2022, at 9:53 PM, PengHui Li <pe...@apache.org> wrote:
> >
> > I think we can use a separate org to maintain the nightly builded
> snapshot
> > images?
> > So that the external repo can pull the latest image from the new org.
> >
> > Regards,
> > Penghui
> >
> >
> >
> >
> >> On Sat, Mar 19, 2022 at 12:33 PM Ziyao Wei
> >> <zi...@streamnative.io.invalid> wrote:
> >>
> >> Hi Pulsar developers,
> >>
> >>
> >> Recently on Docker Hub the `latest` image has been changed from 2.8.2 to
> >> 2.9.1. As a result, the Golang client CI is failing. I have been
> debugging
> >> the issue, but to help timely detection of similar issues in the future
> and
> >> to ease debugging, I propose we should create a nightly Docker image for
> >> Pulsar builds.
> >>
> >> I filed https://github.com/apache/pulsar/issues/14755, and the
> arguments
> >> are as follows:
> >>
> >> *Is your enhancement request related to a problem? Please describe.*
> >>
> >> Currently Docker Hub only contains release images for Pulsar. This makes
> >> doing CI for dependent repos harder, since they wouldn't be able to test
> >> again the bleeding edge easily, and has led to issues such as
> >> https://github.com/apache/pulsar-client-go/issues/748 where it's
> unclear
> >> which Pulsar change caused the issue.
> >>
> >> (The linked issue was made worse because the latest image has been
> outdated
> >> by ~6 months, but 1) even if that's not the case we'd still only be
> able to
> >> catch issues between releases, and 2) the fact that the Golang client is
> >> using latest points to a need of having a nightly.)
> >>
> >> *Describe the solution you'd **like*
> >>
> >> Create a nightly build so external repos can always point against it and
> >> catch issues as early as possible.
> >>
> >> *Describe alternatives you've considered*
> >>
> >> The external repos can also try to pull in the latest Pulsar code, but
> that
> >> seems harder and needs more per repo configuration.
> >>
> >> @eolivelli suggested I post this here so it can be discussed. Opinions
> and
> >> ideas are welcome, thanks!
> >>
> >>
> >> Best,
> >>
> >> Ziyao
> >>
>
>
Re: [DISCUSS] Create a nightly docker image
Posted by Dave Fisher <wa...@comcast.net>.
The ASF has a JFrog artifactory instance that could help. Ask on builds@apache.org!
Regards,
Dave
Sent from my iPhone
> On Mar 18, 2022, at 9:53 PM, PengHui Li <pe...@apache.org> wrote:
>
> I think we can use a separate org to maintain the nightly builded snapshot
> images?
> So that the external repo can pull the latest image from the new org.
>
> Regards,
> Penghui
>
>
>
>
>> On Sat, Mar 19, 2022 at 12:33 PM Ziyao Wei
>> <zi...@streamnative.io.invalid> wrote:
>>
>> Hi Pulsar developers,
>>
>>
>> Recently on Docker Hub the `latest` image has been changed from 2.8.2 to
>> 2.9.1. As a result, the Golang client CI is failing. I have been debugging
>> the issue, but to help timely detection of similar issues in the future and
>> to ease debugging, I propose we should create a nightly Docker image for
>> Pulsar builds.
>>
>> I filed https://github.com/apache/pulsar/issues/14755, and the arguments
>> are as follows:
>>
>> *Is your enhancement request related to a problem? Please describe.*
>>
>> Currently Docker Hub only contains release images for Pulsar. This makes
>> doing CI for dependent repos harder, since they wouldn't be able to test
>> again the bleeding edge easily, and has led to issues such as
>> https://github.com/apache/pulsar-client-go/issues/748 where it's unclear
>> which Pulsar change caused the issue.
>>
>> (The linked issue was made worse because the latest image has been outdated
>> by ~6 months, but 1) even if that's not the case we'd still only be able to
>> catch issues between releases, and 2) the fact that the Golang client is
>> using latest points to a need of having a nightly.)
>>
>> *Describe the solution you'd **like*
>>
>> Create a nightly build so external repos can always point against it and
>> catch issues as early as possible.
>>
>> *Describe alternatives you've considered*
>>
>> The external repos can also try to pull in the latest Pulsar code, but that
>> seems harder and needs more per repo configuration.
>>
>> @eolivelli suggested I post this here so it can be discussed. Opinions and
>> ideas are welcome, thanks!
>>
>>
>> Best,
>>
>> Ziyao
>>
Re: [DISCUSS] Create a nightly docker image
Posted by PengHui Li <pe...@apache.org>.
I think we can use a separate org to maintain the nightly builded snapshot
images?
So that the external repo can pull the latest image from the new org.
Regards,
Penghui
On Sat, Mar 19, 2022 at 12:33 PM Ziyao Wei
<zi...@streamnative.io.invalid> wrote:
> Hi Pulsar developers,
>
>
> Recently on Docker Hub the `latest` image has been changed from 2.8.2 to
> 2.9.1. As a result, the Golang client CI is failing. I have been debugging
> the issue, but to help timely detection of similar issues in the future and
> to ease debugging, I propose we should create a nightly Docker image for
> Pulsar builds.
>
> I filed https://github.com/apache/pulsar/issues/14755, and the arguments
> are as follows:
>
> *Is your enhancement request related to a problem? Please describe.*
>
> Currently Docker Hub only contains release images for Pulsar. This makes
> doing CI for dependent repos harder, since they wouldn't be able to test
> again the bleeding edge easily, and has led to issues such as
> https://github.com/apache/pulsar-client-go/issues/748 where it's unclear
> which Pulsar change caused the issue.
>
> (The linked issue was made worse because the latest image has been outdated
> by ~6 months, but 1) even if that's not the case we'd still only be able to
> catch issues between releases, and 2) the fact that the Golang client is
> using latest points to a need of having a nightly.)
>
> *Describe the solution you'd **like*
>
> Create a nightly build so external repos can always point against it and
> catch issues as early as possible.
>
> *Describe alternatives you've considered*
>
> The external repos can also try to pull in the latest Pulsar code, but that
> seems harder and needs more per repo configuration.
>
> @eolivelli suggested I post this here so it can be discussed. Opinions and
> ideas are welcome, thanks!
>
>
> Best,
>
> Ziyao
>