You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Don Bowman <do...@agilicus.com> on 2019/03/05 16:20:09 UTC
Re: docker build
where do we stand on this?
the PR is in and accepted, but i feel we need to have this built as part of
the release artifacts and on dockerhub to foster adoption.
if the only issue is the mysql connector i can remove it in favour of the
postgres connector.
On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> i can just remove the mysql, the postgres works, i was just assuming folks
> wanted it.
>
>
> On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
>
>> A discussion is progressing on
>> https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to have
>> got anywhere firm yet.
>>
>> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > I don't think anything is strictly needed from you at this point, but
>> > things happen when people drive them, and participation in that effort
>> > would help make sure it gets done. I think at this point the tasks on
>> our
>> > end are watching LEGAL-437 for advice (or making it moot by removing the
>> > MySQL jar), asking Infra to set up automated builds once that is sorted
>> > out, and building some kind of consensus around how we'll label and
>> promote
>> > the Docker images.
>> >
>> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
>> >
>> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
>> metadata.
>> >> if this is the case we should consider relfecting postgres as the
>> default
>> >> metadata in the docs.
>> >> however, i think this is mere aggregation under the gpl license, and
>> the
>> >> docker image tends to have other (e.g. bash) gpl code. druid's start
>> >> scripts are all bash-specific as an example.
>> >>
>> >> I'm not clear if anything further is needed of me, i'm hoping to get an
>> >> automated build going into dockerhub, and tagged w/ each release. i
>> think
>> >> this will help adoption.
>> >>
>> >>
>> >>
>> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
>> >>
>> >> > First off thanks a lot for your work here Don!!
>> >> >
>> >> > I really do think, though, that we need to be careful about the
>> >> inclusion
>> >> > of the MySQL connector jar. ASF legal has been clear in the past that
>> >> ASF
>> >> > projects should not distribute it as part of binary convenience
>> >> releases:
>> >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
>> >> > Dockerfile in the repo is probably fine: in that case we are not
>> >> > distributing the jar itself, just, essentially, a pointer to how to
>> >> > download it. But if we start offering a prebuilt Docker image, it is
>> >> less
>> >> > clear to me if that is fine or not. In the interests of resolving
>> this
>> >> > question one way or the other, I opened a question asking about this
>> >> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
>> >> >
>> >> > About Dylan's questions: my feeling is that we should go ahead and
>> >> enable
>> >> > automated pushes to Docker Hub, and provide some appropriate language
>> >> > around what people should expect out of it. I don't think
>> >> 'experimental' is
>> >> > the right word, but we should be clear around exactly what contract
>> we
>> >> are
>> >> > adhering to. Is it something people can expect to be published with
>> each
>> >> > release? Is it something that we are going to build and test as part
>> of
>> >> the
>> >> > release process, or are we going to publish it via automation without
>> >> any
>> >> > testing? Is it something we expect people to use in production, or
>> >> > something we only expect people to use for evaluation? If it is
>> >> something
>> >> > we expect people to use in production, do we expect them to use it in
>> >> any
>> >> > particular way? Will we be guaranteeing certain things (file layout,
>> >> etc)
>> >> > that provide a stable interface for people to build derived images
>> from?
>> >> >
>> >> > The path of least resistance to answering these questions is to say
>> that
>> >> > the Docker image is provided in the hopes that it is useful, but that
>> >> it is
>> >> > done via an automated build, without any pre-release testing, and
>> >> without
>> >> > any particular guarantees about the 'interface' it provides. If this
>> is
>> >> the
>> >> > case then I would suggest putting it up on Docker Hub with an
>> >> appropriate
>> >> > disclaimer and not promoting it too much. (We might very well end up
>> >> > pushing images every once in a while that don't work right, and it
>> would
>> >> > reflect poorly on the project to have those be prominently
>> linked-to.)
>> >> It
>> >> > becomes easier to strengthen these guarantees if we add an automated
>> >> test
>> >> > suite that we can run before releases which verifies functionality
>> and
>> >> > interface adherence.
>> >> >
>> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
>> >> <rm...@vmware.com.invalid>
>> >> > wrote:
>> >> >
>> >> > > This is purely a packaging exercise. I don't see a reason to mark
>> >> this as
>> >> > > experimental.
>> >> > >
>> >> > > Rajiv.
>> >> > > ________________________________
>> >> > > From: Dylan Wylie <dy...@apache.org>
>> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
>> >> > > To: dev@druid.apache.org
>> >> > > Subject: Re: docker build
>> >> > >
>> >> > > I believe all we have to do is submit a ticket to Apache's
>> >> Infrastructure
>> >> > > team and then we'll have some automatic process that'll
>> automatically
>> >> > > update docker-hub with images relating to each release.
>> >> > >
>> >> > > I guess there's two open questions I think we should reach a
>> >> consensus on
>> >> > > (others feel free to add more!).
>> >> > >
>> >> > > - Are we as a community happy to "support" an additional release
>> >> > artefact?
>> >> > > I'm happy to try to incorporate this into my employer's testing
>> >> > > infrastructure to help catch any regressions on future releases but
>> >> > that's
>> >> > > just one data point on each release.
>> >> > >
>> >> > > - Along the same vein, do we follow the same process as we do with
>> new
>> >> > > features and mark this as experimental for some time?
>> >> > >
>> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
>> >> > >
>> >> > > > Now that
>> >> > >
>> >> >
>> >>
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
>> >> > > is merged
>> >> > > > (thank you!)
>> >> > > >
>> >> > > > who can get this set to build into Dockerhub? Presumably
>> >> automatically
>> >> > > on a
>> >> > > > 'tag' of the repo.
>> >> > > >
>> >> > > > Once that is done it is much more convenient for folks to use
>> this
>> >> > tool.
>> >> > > >
>> >> > > > --don
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
David, that plan looks good to me too. Is anyone interested in implementing
it (maybe David or Don?). Seems to me like the steps needed would be,
1) Do a PR to implement the Dockerfile changes in the Druid repo (remove
MySQL from the Dockerfile that will be used for DockerHub).
2) Work with Infra to set up automated tag builds. Examples -
https://jira.apache.org/jira/browse/INFRA-13838,
https://jira.apache.org/jira/browse/INFRA-12781.
3) Potentially add another Dockerfile FROM the autobuilt image that adds
MySQL and that interested users could run on their own.
On Tue, Mar 5, 2019 at 11:19 AM Charles Allen
<ch...@snap.com.invalid> wrote:
> I would support that.
>
> On Tue, Mar 5, 2019 at 11:13 AM David Glasser <gl...@apollographql.com>
> wrote:
>
> > Is the following a reasonable solution from both a usability and
> > legal perspective?
> > - Write a Dockerfile that has everything except the GPL jar on it
> > (including the Druid code that talks to the jar if configured to use
> MySQL)
> > - Automatically publish that Docker image to the ASF account on DockerHub
> > - Also include a short Dockerfile in the repo that starts `FROM` our
> > auto-built account and has a line or two of wget to download the jar
> > (similar to the wget currently in the Dockerfile)
> > - Tell users who want to use MySQL that they must publish that extra
> > layered image themselves
> >
> > --dave
> >
> >
> > On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <charles.allen@snap.com
> > .invalid>
> > wrote:
> >
> > > Honestly we're at a very strange impasse. On one hand I don't think the
> > ASF
> > > project can adopt an official docker image unless ASF legal says its
> ok.
> > > "Official" releases are source code anyways (as my understanding goes),
> > and
> > > binary artifacts are convenience things. Unfortunately I do not see a
> > path
> > > forward unless some entity is willing to take on a stance similar as
> > > outlined in
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=x7yglOWgfT8oqHqcgn_BnihcsuEZ-gE5QzWksV1PCDg&s=XPfeziOkU4SCezu5EyBUj-pEGcxOY7XVHU1rdNobwrw&e=
> > . This is
> > > pretty new territory from a legal perspective (the fact that docker
> > images
> > > are layers makes it even more interesting).
> > >
> > > At this point I think the safest thing to do is something that is "no
> > more
> > > GPL dependent than other containers in the apache repo", which would
> mean
> > > not adding in GPL binaries. Which means switching to postgres. I don't
> > > foresee an aggressive legal stance on this issue, meaning it might
> take a
> > > while as people watch where the industry is going.
> > >
> > >
> > >
> > > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > where do we stand on this?
> > > > the PR is in and accepted, but i feel we need to have this built as
> > part
> > > of
> > > > the release artifacts and on dockerhub to foster adoption.
> > > > if the only issue is the mysql connector i can remove it in favour of
> > the
> > > > postgres connector.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > > >
> > > > > i can just remove the mysql, the postgres works, i was just
> assuming
> > > > folks
> > > > > wanted it.
> > > > >
> > > > >
> > > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org>
> wrote:
> > > > >
> > > > >> A discussion is progressing on
> > > > >>
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > > It doesn't seem to have
> > > > >> got anywhere firm yet.
> > > > >>
> > > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >>
> > > > >> > I don't think anything is strictly needed from you at this
> point,
> > > but
> > > > >> > things happen when people drive them, and participation in that
> > > effort
> > > > >> > would help make sure it gets done. I think at this point the
> tasks
> > > on
> > > > >> our
> > > > >> > end are watching LEGAL-437 for advice (or making it moot by
> > removing
> > > > the
> > > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > > sorted
> > > > >> > out, and building some kind of consensus around how we'll label
> > and
> > > > >> promote
> > > > >> > the Docker images.
> > > > >> >
> > > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > > wrote:
> > > > >> >
> > > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > > >> metadata.
> > > > >> >> if this is the case we should consider relfecting postgres as
> the
> > > > >> default
> > > > >> >> metadata in the docs.
> > > > >> >> however, i think this is mere aggregation under the gpl
> license,
> > > and
> > > > >> the
> > > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > > start
> > > > >> >> scripts are all bash-specific as an example.
> > > > >> >>
> > > > >> >> I'm not clear if anything further is needed of me, i'm hoping
> to
> > > get
> > > > an
> > > > >> >> automated build going into dockerhub, and tagged w/ each
> > release. i
> > > > >> think
> > > > >> >> this will help adoption.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >> >>
> > > > >> >> > First off thanks a lot for your work here Don!!
> > > > >> >> >
> > > > >> >> > I really do think, though, that we need to be careful about
> the
> > > > >> >> inclusion
> > > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> > past
> > > > that
> > > > >> >> ASF
> > > > >> >> > projects should not distribute it as part of binary
> convenience
> > > > >> >> releases:
> > > > >> >> >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > > .
> > > > I think having the
> > > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> > not
> > > > >> >> > distributing the jar itself, just, essentially, a pointer to
> > how
> > > to
> > > > >> >> > download it. But if we start offering a prebuilt Docker
> image,
> > it
> > > > is
> > > > >> >> less
> > > > >> >> > clear to me if that is fine or not. In the interests of
> > resolving
> > > > >> this
> > > > >> >> > question one way or the other, I opened a question asking
> about
> > > > this
> > > > >> >> > specific situation:
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > > .
> > > > >> >> >
> > > > >> >> > About Dylan's questions: my feeling is that we should go
> ahead
> > > and
> > > > >> >> enable
> > > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > > language
> > > > >> >> > around what people should expect out of it. I don't think
> > > > >> >> 'experimental' is
> > > > >> >> > the right word, but we should be clear around exactly what
> > > contract
> > > > >> we
> > > > >> >> are
> > > > >> >> > adhering to. Is it something people can expect to be
> published
> > > with
> > > > >> each
> > > > >> >> > release? Is it something that we are going to build and test
> as
> > > > part
> > > > >> of
> > > > >> >> the
> > > > >> >> > release process, or are we going to publish it via automation
> > > > without
> > > > >> >> any
> > > > >> >> > testing? Is it something we expect people to use in
> production,
> > > or
> > > > >> >> > something we only expect people to use for evaluation? If it
> is
> > > > >> >> something
> > > > >> >> > we expect people to use in production, do we expect them to
> use
> > > it
> > > > in
> > > > >> >> any
> > > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > > layout,
> > > > >> >> etc)
> > > > >> >> > that provide a stable interface for people to build derived
> > > images
> > > > >> from?
> > > > >> >> >
> > > > >> >> > The path of least resistance to answering these questions is
> to
> > > say
> > > > >> that
> > > > >> >> > the Docker image is provided in the hopes that it is useful,
> > but
> > > > that
> > > > >> >> it is
> > > > >> >> > done via an automated build, without any pre-release testing,
> > and
> > > > >> >> without
> > > > >> >> > any particular guarantees about the 'interface' it provides.
> If
> > > > this
> > > > >> is
> > > > >> >> the
> > > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > > >> >> appropriate
> > > > >> >> > disclaimer and not promoting it too much. (We might very well
> > end
> > > > up
> > > > >> >> > pushing images every once in a while that don't work right,
> and
> > > it
> > > > >> would
> > > > >> >> > reflect poorly on the project to have those be prominently
> > > > >> linked-to.)
> > > > >> >> It
> > > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > > automated
> > > > >> >> test
> > > > >> >> > suite that we can run before releases which verifies
> > > functionality
> > > > >> and
> > > > >> >> > interface adherence.
> > > > >> >> >
> > > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > > >> >> <rm...@vmware.com.invalid>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> > > This is purely a packaging exercise. I don't see a reason
> to
> > > mark
> > > > >> >> this as
> > > > >> >> > > experimental.
> > > > >> >> > >
> > > > >> >> > > Rajiv.
> > > > >> >> > > ________________________________
> > > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > > >> >> > > To: dev@druid.apache.org
> > > > >> >> > > Subject: Re: docker build
> > > > >> >> > >
> > > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > > >> >> Infrastructure
> > > > >> >> > > team and then we'll have some automatic process that'll
> > > > >> automatically
> > > > >> >> > > update docker-hub with images relating to each release.
> > > > >> >> > >
> > > > >> >> > > I guess there's two open questions I think we should reach
> a
> > > > >> >> consensus on
> > > > >> >> > > (others feel free to add more!).
> > > > >> >> > >
> > > > >> >> > > - Are we as a community happy to "support" an additional
> > > release
> > > > >> >> > artefact?
> > > > >> >> > > I'm happy to try to incorporate this into my employer's
> > testing
> > > > >> >> > > infrastructure to help catch any regressions on future
> > releases
> > > > but
> > > > >> >> > that's
> > > > >> >> > > just one data point on each release.
> > > > >> >> > >
> > > > >> >> > > - Along the same vein, do we follow the same process as we
> do
> > > > with
> > > > >> new
> > > > >> >> > > features and mark this as experimental for some time?
> > > > >> >> > >
> > > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > > wrote:
> > > > >> >> > >
> > > > >> >> > > > Now that
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > > >> >> > > is merged
> > > > >> >> > > > (thank you!)
> > > > >> >> > > >
> > > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > > >> >> automatically
> > > > >> >> > > on a
> > > > >> >> > > > 'tag' of the repo.
> > > > >> >> > > >
> > > > >> >> > > > Once that is done it is much more convenient for folks to
> > use
> > > > >> this
> > > > >> >> > tool.
> > > > >> >> > > >
> > > > >> >> > > > --don
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
I would support that.
On Tue, Mar 5, 2019 at 11:13 AM David Glasser <gl...@apollographql.com>
wrote:
> Is the following a reasonable solution from both a usability and
> legal perspective?
> - Write a Dockerfile that has everything except the GPL jar on it
> (including the Druid code that talks to the jar if configured to use MySQL)
> - Automatically publish that Docker image to the ASF account on DockerHub
> - Also include a short Dockerfile in the repo that starts `FROM` our
> auto-built account and has a line or two of wget to download the jar
> (similar to the wget currently in the Dockerfile)
> - Tell users who want to use MySQL that they must publish that extra
> layered image themselves
>
> --dave
>
>
> On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <charles.allen@snap.com
> .invalid>
> wrote:
>
> > Honestly we're at a very strange impasse. On one hand I don't think the
> ASF
> > project can adopt an official docker image unless ASF legal says its ok.
> > "Official" releases are source code anyways (as my understanding goes),
> and
> > binary artifacts are convenience things. Unfortunately I do not see a
> path
> > forward unless some entity is willing to take on a stance similar as
> > outlined in
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=x7yglOWgfT8oqHqcgn_BnihcsuEZ-gE5QzWksV1PCDg&s=XPfeziOkU4SCezu5EyBUj-pEGcxOY7XVHU1rdNobwrw&e=
> . This is
> > pretty new territory from a legal perspective (the fact that docker
> images
> > are layers makes it even more interesting).
> >
> > At this point I think the safest thing to do is something that is "no
> more
> > GPL dependent than other containers in the apache repo", which would mean
> > not adding in GPL binaries. Which means switching to postgres. I don't
> > foresee an aggressive legal stance on this issue, meaning it might take a
> > while as people watch where the industry is going.
> >
> >
> >
> > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> >
> > > where do we stand on this?
> > > the PR is in and accepted, but i feel we need to have this built as
> part
> > of
> > > the release artifacts and on dockerhub to foster adoption.
> > > if the only issue is the mysql connector i can remove it in favour of
> the
> > > postgres connector.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > i can just remove the mysql, the postgres works, i was just assuming
> > > folks
> > > > wanted it.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > >> A discussion is progressing on
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > > It doesn't seem to have
> > > >> got anywhere firm yet.
> > > >>
> > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >>
> > > >> > I don't think anything is strictly needed from you at this point,
> > but
> > > >> > things happen when people drive them, and participation in that
> > effort
> > > >> > would help make sure it gets done. I think at this point the tasks
> > on
> > > >> our
> > > >> > end are watching LEGAL-437 for advice (or making it moot by
> removing
> > > the
> > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > sorted
> > > >> > out, and building some kind of consensus around how we'll label
> and
> > > >> promote
> > > >> > the Docker images.
> > > >> >
> > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > wrote:
> > > >> >
> > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > >> metadata.
> > > >> >> if this is the case we should consider relfecting postgres as the
> > > >> default
> > > >> >> metadata in the docs.
> > > >> >> however, i think this is mere aggregation under the gpl license,
> > and
> > > >> the
> > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > start
> > > >> >> scripts are all bash-specific as an example.
> > > >> >>
> > > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> > get
> > > an
> > > >> >> automated build going into dockerhub, and tagged w/ each
> release. i
> > > >> think
> > > >> >> this will help adoption.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> wrote:
> > > >> >>
> > > >> >> > First off thanks a lot for your work here Don!!
> > > >> >> >
> > > >> >> > I really do think, though, that we need to be careful about the
> > > >> >> inclusion
> > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> past
> > > that
> > > >> >> ASF
> > > >> >> > projects should not distribute it as part of binary convenience
> > > >> >> releases:
> > > >> >> >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > .
> > > I think having the
> > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> not
> > > >> >> > distributing the jar itself, just, essentially, a pointer to
> how
> > to
> > > >> >> > download it. But if we start offering a prebuilt Docker image,
> it
> > > is
> > > >> >> less
> > > >> >> > clear to me if that is fine or not. In the interests of
> resolving
> > > >> this
> > > >> >> > question one way or the other, I opened a question asking about
> > > this
> > > >> >> > specific situation:
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > >> >> >
> > > >> >> > About Dylan's questions: my feeling is that we should go ahead
> > and
> > > >> >> enable
> > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > language
> > > >> >> > around what people should expect out of it. I don't think
> > > >> >> 'experimental' is
> > > >> >> > the right word, but we should be clear around exactly what
> > contract
> > > >> we
> > > >> >> are
> > > >> >> > adhering to. Is it something people can expect to be published
> > with
> > > >> each
> > > >> >> > release? Is it something that we are going to build and test as
> > > part
> > > >> of
> > > >> >> the
> > > >> >> > release process, or are we going to publish it via automation
> > > without
> > > >> >> any
> > > >> >> > testing? Is it something we expect people to use in production,
> > or
> > > >> >> > something we only expect people to use for evaluation? If it is
> > > >> >> something
> > > >> >> > we expect people to use in production, do we expect them to use
> > it
> > > in
> > > >> >> any
> > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > layout,
> > > >> >> etc)
> > > >> >> > that provide a stable interface for people to build derived
> > images
> > > >> from?
> > > >> >> >
> > > >> >> > The path of least resistance to answering these questions is to
> > say
> > > >> that
> > > >> >> > the Docker image is provided in the hopes that it is useful,
> but
> > > that
> > > >> >> it is
> > > >> >> > done via an automated build, without any pre-release testing,
> and
> > > >> >> without
> > > >> >> > any particular guarantees about the 'interface' it provides. If
> > > this
> > > >> is
> > > >> >> the
> > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > >> >> appropriate
> > > >> >> > disclaimer and not promoting it too much. (We might very well
> end
> > > up
> > > >> >> > pushing images every once in a while that don't work right, and
> > it
> > > >> would
> > > >> >> > reflect poorly on the project to have those be prominently
> > > >> linked-to.)
> > > >> >> It
> > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > automated
> > > >> >> test
> > > >> >> > suite that we can run before releases which verifies
> > functionality
> > > >> and
> > > >> >> > interface adherence.
> > > >> >> >
> > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > >> >> <rm...@vmware.com.invalid>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > This is purely a packaging exercise. I don't see a reason to
> > mark
> > > >> >> this as
> > > >> >> > > experimental.
> > > >> >> > >
> > > >> >> > > Rajiv.
> > > >> >> > > ________________________________
> > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > >> >> > > To: dev@druid.apache.org
> > > >> >> > > Subject: Re: docker build
> > > >> >> > >
> > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > >> >> Infrastructure
> > > >> >> > > team and then we'll have some automatic process that'll
> > > >> automatically
> > > >> >> > > update docker-hub with images relating to each release.
> > > >> >> > >
> > > >> >> > > I guess there's two open questions I think we should reach a
> > > >> >> consensus on
> > > >> >> > > (others feel free to add more!).
> > > >> >> > >
> > > >> >> > > - Are we as a community happy to "support" an additional
> > release
> > > >> >> > artefact?
> > > >> >> > > I'm happy to try to incorporate this into my employer's
> testing
> > > >> >> > > infrastructure to help catch any regressions on future
> releases
> > > but
> > > >> >> > that's
> > > >> >> > > just one data point on each release.
> > > >> >> > >
> > > >> >> > > - Along the same vein, do we follow the same process as we do
> > > with
> > > >> new
> > > >> >> > > features and mark this as experimental for some time?
> > > >> >> > >
> > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > wrote:
> > > >> >> > >
> > > >> >> > > > Now that
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > >> >> > > is merged
> > > >> >> > > > (thank you!)
> > > >> >> > > >
> > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > >> >> automatically
> > > >> >> > > on a
> > > >> >> > > > 'tag' of the repo.
> > > >> >> > > >
> > > >> >> > > > Once that is done it is much more convenient for folks to
> use
> > > >> this
> > > >> >> > tool.
> > > >> >> > > >
> > > >> >> > > > --don
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: docker build
Posted by David Glasser <gl...@apollographql.com>.
Is the following a reasonable solution from both a usability and
legal perspective?
- Write a Dockerfile that has everything except the GPL jar on it
(including the Druid code that talks to the jar if configured to use MySQL)
- Automatically publish that Docker image to the ASF account on DockerHub
- Also include a short Dockerfile in the repo that starts `FROM` our
auto-built account and has a line or two of wget to download the jar
(similar to the wget currently in the Dockerfile)
- Tell users who want to use MySQL that they must publish that extra
layered image themselves
--dave
On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <ch...@snap.com.invalid>
wrote:
> Honestly we're at a very strange impasse. On one hand I don't think the ASF
> project can adopt an official docker image unless ASF legal says its ok.
> "Official" releases are source code anyways (as my understanding goes), and
> binary artifacts are convenience things. Unfortunately I do not see a path
> forward unless some entity is willing to take on a stance similar as
> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> pretty new territory from a legal perspective (the fact that docker images
> are layers makes it even more interesting).
>
> At this point I think the safest thing to do is something that is "no more
> GPL dependent than other containers in the apache repo", which would mean
> not adding in GPL binaries. Which means switching to postgres. I don't
> foresee an aggressive legal stance on this issue, meaning it might take a
> while as people watch where the industry is going.
>
>
>
> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>
> > where do we stand on this?
> > the PR is in and accepted, but i feel we need to have this built as part
> of
> > the release artifacts and on dockerhub to foster adoption.
> > if the only issue is the mysql connector i can remove it in favour of the
> > postgres connector.
> >
> >
> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> >
> > > i can just remove the mysql, the postgres works, i was just assuming
> > folks
> > > wanted it.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> A discussion is progressing on
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> > It doesn't seem to have
> > >> got anywhere firm yet.
> > >>
> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > I don't think anything is strictly needed from you at this point,
> but
> > >> > things happen when people drive them, and participation in that
> effort
> > >> > would help make sure it gets done. I think at this point the tasks
> on
> > >> our
> > >> > end are watching LEGAL-437 for advice (or making it moot by removing
> > the
> > >> > MySQL jar), asking Infra to set up automated builds once that is
> > sorted
> > >> > out, and building some kind of consensus around how we'll label and
> > >> promote
> > >> > the Docker images.
> > >> >
> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> wrote:
> > >> >
> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > >> metadata.
> > >> >> if this is the case we should consider relfecting postgres as the
> > >> default
> > >> >> metadata in the docs.
> > >> >> however, i think this is mere aggregation under the gpl license,
> and
> > >> the
> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> start
> > >> >> scripts are all bash-specific as an example.
> > >> >>
> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> get
> > an
> > >> >> automated build going into dockerhub, and tagged w/ each release. i
> > >> think
> > >> >> this will help adoption.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > >> >>
> > >> >> > First off thanks a lot for your work here Don!!
> > >> >> >
> > >> >> > I really do think, though, that we need to be careful about the
> > >> >> inclusion
> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> > that
> > >> >> ASF
> > >> >> > projects should not distribute it as part of binary convenience
> > >> >> releases:
> > >> >> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> .
> > I think having the
> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
> > >> >> > distributing the jar itself, just, essentially, a pointer to how
> to
> > >> >> > download it. But if we start offering a prebuilt Docker image, it
> > is
> > >> >> less
> > >> >> > clear to me if that is fine or not. In the interests of resolving
> > >> this
> > >> >> > question one way or the other, I opened a question asking about
> > this
> > >> >> > specific situation:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > >> >> >
> > >> >> > About Dylan's questions: my feeling is that we should go ahead
> and
> > >> >> enable
> > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > language
> > >> >> > around what people should expect out of it. I don't think
> > >> >> 'experimental' is
> > >> >> > the right word, but we should be clear around exactly what
> contract
> > >> we
> > >> >> are
> > >> >> > adhering to. Is it something people can expect to be published
> with
> > >> each
> > >> >> > release? Is it something that we are going to build and test as
> > part
> > >> of
> > >> >> the
> > >> >> > release process, or are we going to publish it via automation
> > without
> > >> >> any
> > >> >> > testing? Is it something we expect people to use in production,
> or
> > >> >> > something we only expect people to use for evaluation? If it is
> > >> >> something
> > >> >> > we expect people to use in production, do we expect them to use
> it
> > in
> > >> >> any
> > >> >> > particular way? Will we be guaranteeing certain things (file
> > layout,
> > >> >> etc)
> > >> >> > that provide a stable interface for people to build derived
> images
> > >> from?
> > >> >> >
> > >> >> > The path of least resistance to answering these questions is to
> say
> > >> that
> > >> >> > the Docker image is provided in the hopes that it is useful, but
> > that
> > >> >> it is
> > >> >> > done via an automated build, without any pre-release testing, and
> > >> >> without
> > >> >> > any particular guarantees about the 'interface' it provides. If
> > this
> > >> is
> > >> >> the
> > >> >> > case then I would suggest putting it up on Docker Hub with an
> > >> >> appropriate
> > >> >> > disclaimer and not promoting it too much. (We might very well end
> > up
> > >> >> > pushing images every once in a while that don't work right, and
> it
> > >> would
> > >> >> > reflect poorly on the project to have those be prominently
> > >> linked-to.)
> > >> >> It
> > >> >> > becomes easier to strengthen these guarantees if we add an
> > automated
> > >> >> test
> > >> >> > suite that we can run before releases which verifies
> functionality
> > >> and
> > >> >> > interface adherence.
> > >> >> >
> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > >> >> <rm...@vmware.com.invalid>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > This is purely a packaging exercise. I don't see a reason to
> mark
> > >> >> this as
> > >> >> > > experimental.
> > >> >> > >
> > >> >> > > Rajiv.
> > >> >> > > ________________________________
> > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > >> >> > > To: dev@druid.apache.org
> > >> >> > > Subject: Re: docker build
> > >> >> > >
> > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > >> >> Infrastructure
> > >> >> > > team and then we'll have some automatic process that'll
> > >> automatically
> > >> >> > > update docker-hub with images relating to each release.
> > >> >> > >
> > >> >> > > I guess there's two open questions I think we should reach a
> > >> >> consensus on
> > >> >> > > (others feel free to add more!).
> > >> >> > >
> > >> >> > > - Are we as a community happy to "support" an additional
> release
> > >> >> > artefact?
> > >> >> > > I'm happy to try to incorporate this into my employer's testing
> > >> >> > > infrastructure to help catch any regressions on future releases
> > but
> > >> >> > that's
> > >> >> > > just one data point on each release.
> > >> >> > >
> > >> >> > > - Along the same vein, do we follow the same process as we do
> > with
> > >> new
> > >> >> > > features and mark this as experimental for some time?
> > >> >> > >
> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > wrote:
> > >> >> > >
> > >> >> > > > Now that
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > >> >> > > is merged
> > >> >> > > > (thank you!)
> > >> >> > > >
> > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > >> >> automatically
> > >> >> > > on a
> > >> >> > > > 'tag' of the repo.
> > >> >> > > >
> > >> >> > > > Once that is done it is much more convenient for folks to use
> > >> this
> > >> >> > tool.
> > >> >> > > >
> > >> >> > > > --don
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
Hey Don,
If you are interested in participating more directly in the Druid project,
you could certainly do #2. Apache projects are built around the idea of
community contribution, and people driving & building consensus on the
changes they want to see happen. Apache Infra is a group of people that
manage the Apache infrastructure: https://www.apache.org/dev/infrastructure.
They set up our mailing lists, git repos, etc, and can also set up
DockerHub builds. They use JIRA to track requests. Anyway, it is up to you.
On Tue, Mar 19, 2019 at 4:19 AM Don Bowman <do...@agilicus.com> wrote:
> So if I send a PR removing the mysql connector (which comes from maven
> which is itself apache released), can we be done w/ this?
>
> I don't want to get into the fact that it runs on linux and uses bash in
> scripts.
>
> I think there's some confusion in this thread about what people expect from
> container images. We want this to be the standard image in the Kubernetes
> helm chart as an exxample. No one wants to buiild their own container and
> find their own registry to host it.
>
> But if its solely about mysql, I think removing it is fine, the postgres
> connector works at least as well.
>
> On the 1/2/3 list:
>
> #1: I'll do the PR to remove mysql
> #2. I cannot do this, I have no idea what Jira or apache infra is etc.
> #3. I don't think this is worth doing, no one will use it except for
> someone who can trivially do that themself and might want other things in
> there anyway.
>
> so, if someone can help w/ #2 (getting the travis build to push this or
> whatever) i'll do a PR for #1.
> OK?
>
> On Tue, 5 Mar 2019 at 12:59, Charles Allen <charles.allen@snap.com
> .invalid>
> wrote:
>
> > Honestly we're at a very strange impasse. On one hand I don't think the
> ASF
> > project can adopt an official docker image unless ASF legal says its ok.
> > "Official" releases are source code anyways (as my understanding goes),
> and
> > binary artifacts are convenience things. Unfortunately I do not see a
> path
> > forward unless some entity is willing to take on a stance similar as
> > outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> > pretty new territory from a legal perspective (the fact that docker
> images
> > are layers makes it even more interesting).
> >
> > At this point I think the safest thing to do is something that is "no
> more
> > GPL dependent than other containers in the apache repo", which would mean
> > not adding in GPL binaries. Which means switching to postgres. I don't
> > foresee an aggressive legal stance on this issue, meaning it might take a
> > while as people watch where the industry is going.
> >
> >
> >
> > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> >
> > > where do we stand on this?
> > > the PR is in and accepted, but i feel we need to have this built as
> part
> > of
> > > the release artifacts and on dockerhub to foster adoption.
> > > if the only issue is the mysql connector i can remove it in favour of
> the
> > > postgres connector.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > i can just remove the mysql, the postgres works, i was just assuming
> > > folks
> > > > wanted it.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > >> A discussion is progressing on
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > > It doesn't seem to have
> > > >> got anywhere firm yet.
> > > >>
> > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >>
> > > >> > I don't think anything is strictly needed from you at this point,
> > but
> > > >> > things happen when people drive them, and participation in that
> > effort
> > > >> > would help make sure it gets done. I think at this point the tasks
> > on
> > > >> our
> > > >> > end are watching LEGAL-437 for advice (or making it moot by
> removing
> > > the
> > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > sorted
> > > >> > out, and building some kind of consensus around how we'll label
> and
> > > >> promote
> > > >> > the Docker images.
> > > >> >
> > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > wrote:
> > > >> >
> > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > >> metadata.
> > > >> >> if this is the case we should consider relfecting postgres as the
> > > >> default
> > > >> >> metadata in the docs.
> > > >> >> however, i think this is mere aggregation under the gpl license,
> > and
> > > >> the
> > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > start
> > > >> >> scripts are all bash-specific as an example.
> > > >> >>
> > > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> > get
> > > an
> > > >> >> automated build going into dockerhub, and tagged w/ each
> release. i
> > > >> think
> > > >> >> this will help adoption.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> wrote:
> > > >> >>
> > > >> >> > First off thanks a lot for your work here Don!!
> > > >> >> >
> > > >> >> > I really do think, though, that we need to be careful about the
> > > >> >> inclusion
> > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> past
> > > that
> > > >> >> ASF
> > > >> >> > projects should not distribute it as part of binary convenience
> > > >> >> releases:
> > > >> >> >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > .
> > > I think having the
> > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> not
> > > >> >> > distributing the jar itself, just, essentially, a pointer to
> how
> > to
> > > >> >> > download it. But if we start offering a prebuilt Docker image,
> it
> > > is
> > > >> >> less
> > > >> >> > clear to me if that is fine or not. In the interests of
> resolving
> > > >> this
> > > >> >> > question one way or the other, I opened a question asking about
> > > this
> > > >> >> > specific situation:
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > >> >> >
> > > >> >> > About Dylan's questions: my feeling is that we should go ahead
> > and
> > > >> >> enable
> > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > language
> > > >> >> > around what people should expect out of it. I don't think
> > > >> >> 'experimental' is
> > > >> >> > the right word, but we should be clear around exactly what
> > contract
> > > >> we
> > > >> >> are
> > > >> >> > adhering to. Is it something people can expect to be published
> > with
> > > >> each
> > > >> >> > release? Is it something that we are going to build and test as
> > > part
> > > >> of
> > > >> >> the
> > > >> >> > release process, or are we going to publish it via automation
> > > without
> > > >> >> any
> > > >> >> > testing? Is it something we expect people to use in production,
> > or
> > > >> >> > something we only expect people to use for evaluation? If it is
> > > >> >> something
> > > >> >> > we expect people to use in production, do we expect them to use
> > it
> > > in
> > > >> >> any
> > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > layout,
> > > >> >> etc)
> > > >> >> > that provide a stable interface for people to build derived
> > images
> > > >> from?
> > > >> >> >
> > > >> >> > The path of least resistance to answering these questions is to
> > say
> > > >> that
> > > >> >> > the Docker image is provided in the hopes that it is useful,
> but
> > > that
> > > >> >> it is
> > > >> >> > done via an automated build, without any pre-release testing,
> and
> > > >> >> without
> > > >> >> > any particular guarantees about the 'interface' it provides. If
> > > this
> > > >> is
> > > >> >> the
> > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > >> >> appropriate
> > > >> >> > disclaimer and not promoting it too much. (We might very well
> end
> > > up
> > > >> >> > pushing images every once in a while that don't work right, and
> > it
> > > >> would
> > > >> >> > reflect poorly on the project to have those be prominently
> > > >> linked-to.)
> > > >> >> It
> > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > automated
> > > >> >> test
> > > >> >> > suite that we can run before releases which verifies
> > functionality
> > > >> and
> > > >> >> > interface adherence.
> > > >> >> >
> > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > >> >> <rm...@vmware.com.invalid>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > This is purely a packaging exercise. I don't see a reason to
> > mark
> > > >> >> this as
> > > >> >> > > experimental.
> > > >> >> > >
> > > >> >> > > Rajiv.
> > > >> >> > > ________________________________
> > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > >> >> > > To: dev@druid.apache.org
> > > >> >> > > Subject: Re: docker build
> > > >> >> > >
> > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > >> >> Infrastructure
> > > >> >> > > team and then we'll have some automatic process that'll
> > > >> automatically
> > > >> >> > > update docker-hub with images relating to each release.
> > > >> >> > >
> > > >> >> > > I guess there's two open questions I think we should reach a
> > > >> >> consensus on
> > > >> >> > > (others feel free to add more!).
> > > >> >> > >
> > > >> >> > > - Are we as a community happy to "support" an additional
> > release
> > > >> >> > artefact?
> > > >> >> > > I'm happy to try to incorporate this into my employer's
> testing
> > > >> >> > > infrastructure to help catch any regressions on future
> releases
> > > but
> > > >> >> > that's
> > > >> >> > > just one data point on each release.
> > > >> >> > >
> > > >> >> > > - Along the same vein, do we follow the same process as we do
> > > with
> > > >> new
> > > >> >> > > features and mark this as experimental for some time?
> > > >> >> > >
> > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > wrote:
> > > >> >> > >
> > > >> >> > > > Now that
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > >> >> > > is merged
> > > >> >> > > > (thank you!)
> > > >> >> > > >
> > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > >> >> automatically
> > > >> >> > > on a
> > > >> >> > > > 'tag' of the repo.
> > > >> >> > > >
> > > >> >> > > > Once that is done it is much more convenient for folks to
> use
> > > >> this
> > > >> >> > tool.
> > > >> >> > > >
> > > >> >> > > > --don
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
https://github.com/apache/incubator-druid/pull/7296 is now there which
removes the mysql but shows how to re-enable should someone wish.
On Tue, 19 Mar 2019 at 07:19, Don Bowman <do...@agilicus.com> wrote:
> So if I send a PR removing the mysql connector (which comes from maven
> which is itself apache released), can we be done w/ this?
>
> I don't want to get into the fact that it runs on linux and uses bash in
> scripts.
>
> I think there's some confusion in this thread about what people expect
> from container images. We want this to be the standard image in the
> Kubernetes helm chart as an exxample. No one wants to buiild their own
> container and find their own registry to host it.
>
> But if its solely about mysql, I think removing it is fine, the postgres
> connector works at least as well.
>
> On the 1/2/3 list:
>
> #1: I'll do the PR to remove mysql
> #2. I cannot do this, I have no idea what Jira or apache infra is etc.
> #3. I don't think this is worth doing, no one will use it except for
> someone who can trivially do that themself and might want other things in
> there anyway.
>
> so, if someone can help w/ #2 (getting the travis build to push this or
> whatever) i'll do a PR for #1.
> OK?
>
> On Tue, 5 Mar 2019 at 12:59, Charles Allen <ch...@snap.com.invalid>
> wrote:
>
>> Honestly we're at a very strange impasse. On one hand I don't think the
>> ASF
>> project can adopt an official docker image unless ASF legal says its ok.
>> "Official" releases are source code anyways (as my understanding goes),
>> and
>> binary artifacts are convenience things. Unfortunately I do not see a path
>> forward unless some entity is willing to take on a stance similar as
>> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
>> pretty new territory from a legal perspective (the fact that docker images
>> are layers makes it even more interesting).
>>
>> At this point I think the safest thing to do is something that is "no more
>> GPL dependent than other containers in the apache repo", which would mean
>> not adding in GPL binaries. Which means switching to postgres. I don't
>> foresee an aggressive legal stance on this issue, meaning it might take a
>> while as people watch where the industry is going.
>>
>>
>>
>> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>>
>> > where do we stand on this?
>> > the PR is in and accepted, but i feel we need to have this built as
>> part of
>> > the release artifacts and on dockerhub to foster adoption.
>> > if the only issue is the mysql connector i can remove it in favour of
>> the
>> > postgres connector.
>> >
>> >
>> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
>> >
>> > > i can just remove the mysql, the postgres works, i was just assuming
>> > folks
>> > > wanted it.
>> > >
>> > >
>> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
>> > >
>> > >> A discussion is progressing on
>> > >>
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
>> .
>> > It doesn't seem to have
>> > >> got anywhere firm yet.
>> > >>
>> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> > >>
>> > >> > I don't think anything is strictly needed from you at this point,
>> but
>> > >> > things happen when people drive them, and participation in that
>> effort
>> > >> > would help make sure it gets done. I think at this point the tasks
>> on
>> > >> our
>> > >> > end are watching LEGAL-437 for advice (or making it moot by
>> removing
>> > the
>> > >> > MySQL jar), asking Infra to set up automated builds once that is
>> > sorted
>> > >> > out, and building some kind of consensus around how we'll label and
>> > >> promote
>> > >> > the Docker images.
>> > >> >
>> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
>> wrote:
>> > >> >
>> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
>> > >> metadata.
>> > >> >> if this is the case we should consider relfecting postgres as the
>> > >> default
>> > >> >> metadata in the docs.
>> > >> >> however, i think this is mere aggregation under the gpl license,
>> and
>> > >> the
>> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
>> start
>> > >> >> scripts are all bash-specific as an example.
>> > >> >>
>> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
>> get
>> > an
>> > >> >> automated build going into dockerhub, and tagged w/ each release.
>> i
>> > >> think
>> > >> >> this will help adoption.
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
>> wrote:
>> > >> >>
>> > >> >> > First off thanks a lot for your work here Don!!
>> > >> >> >
>> > >> >> > I really do think, though, that we need to be careful about the
>> > >> >> inclusion
>> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
>> > that
>> > >> >> ASF
>> > >> >> > projects should not distribute it as part of binary convenience
>> > >> >> releases:
>> > >> >> >
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
>> .
>> > I think having the
>> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
>> > >> >> > distributing the jar itself, just, essentially, a pointer to
>> how to
>> > >> >> > download it. But if we start offering a prebuilt Docker image,
>> it
>> > is
>> > >> >> less
>> > >> >> > clear to me if that is fine or not. In the interests of
>> resolving
>> > >> this
>> > >> >> > question one way or the other, I opened a question asking about
>> > this
>> > >> >> > specific situation:
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
>> > .
>> > >> >> >
>> > >> >> > About Dylan's questions: my feeling is that we should go ahead
>> and
>> > >> >> enable
>> > >> >> > automated pushes to Docker Hub, and provide some appropriate
>> > language
>> > >> >> > around what people should expect out of it. I don't think
>> > >> >> 'experimental' is
>> > >> >> > the right word, but we should be clear around exactly what
>> contract
>> > >> we
>> > >> >> are
>> > >> >> > adhering to. Is it something people can expect to be published
>> with
>> > >> each
>> > >> >> > release? Is it something that we are going to build and test as
>> > part
>> > >> of
>> > >> >> the
>> > >> >> > release process, or are we going to publish it via automation
>> > without
>> > >> >> any
>> > >> >> > testing? Is it something we expect people to use in production,
>> or
>> > >> >> > something we only expect people to use for evaluation? If it is
>> > >> >> something
>> > >> >> > we expect people to use in production, do we expect them to use
>> it
>> > in
>> > >> >> any
>> > >> >> > particular way? Will we be guaranteeing certain things (file
>> > layout,
>> > >> >> etc)
>> > >> >> > that provide a stable interface for people to build derived
>> images
>> > >> from?
>> > >> >> >
>> > >> >> > The path of least resistance to answering these questions is to
>> say
>> > >> that
>> > >> >> > the Docker image is provided in the hopes that it is useful, but
>> > that
>> > >> >> it is
>> > >> >> > done via an automated build, without any pre-release testing,
>> and
>> > >> >> without
>> > >> >> > any particular guarantees about the 'interface' it provides. If
>> > this
>> > >> is
>> > >> >> the
>> > >> >> > case then I would suggest putting it up on Docker Hub with an
>> > >> >> appropriate
>> > >> >> > disclaimer and not promoting it too much. (We might very well
>> end
>> > up
>> > >> >> > pushing images every once in a while that don't work right, and
>> it
>> > >> would
>> > >> >> > reflect poorly on the project to have those be prominently
>> > >> linked-to.)
>> > >> >> It
>> > >> >> > becomes easier to strengthen these guarantees if we add an
>> > automated
>> > >> >> test
>> > >> >> > suite that we can run before releases which verifies
>> functionality
>> > >> and
>> > >> >> > interface adherence.
>> > >> >> >
>> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
>> > >> >> <rm...@vmware.com.invalid>
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > This is purely a packaging exercise. I don't see a reason to
>> mark
>> > >> >> this as
>> > >> >> > > experimental.
>> > >> >> > >
>> > >> >> > > Rajiv.
>> > >> >> > > ________________________________
>> > >> >> > > From: Dylan Wylie <dy...@apache.org>
>> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
>> > >> >> > > To: dev@druid.apache.org
>> > >> >> > > Subject: Re: docker build
>> > >> >> > >
>> > >> >> > > I believe all we have to do is submit a ticket to Apache's
>> > >> >> Infrastructure
>> > >> >> > > team and then we'll have some automatic process that'll
>> > >> automatically
>> > >> >> > > update docker-hub with images relating to each release.
>> > >> >> > >
>> > >> >> > > I guess there's two open questions I think we should reach a
>> > >> >> consensus on
>> > >> >> > > (others feel free to add more!).
>> > >> >> > >
>> > >> >> > > - Are we as a community happy to "support" an additional
>> release
>> > >> >> > artefact?
>> > >> >> > > I'm happy to try to incorporate this into my employer's
>> testing
>> > >> >> > > infrastructure to help catch any regressions on future
>> releases
>> > but
>> > >> >> > that's
>> > >> >> > > just one data point on each release.
>> > >> >> > >
>> > >> >> > > - Along the same vein, do we follow the same process as we do
>> > with
>> > >> new
>> > >> >> > > features and mark this as experimental for some time?
>> > >> >> > >
>> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
>> > wrote:
>> > >> >> > >
>> > >> >> > > > Now that
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
>> > >> >> > > is merged
>> > >> >> > > > (thank you!)
>> > >> >> > > >
>> > >> >> > > > who can get this set to build into Dockerhub? Presumably
>> > >> >> automatically
>> > >> >> > > on a
>> > >> >> > > > 'tag' of the repo.
>> > >> >> > > >
>> > >> >> > > > Once that is done it is much more convenient for folks to
>> use
>> > >> this
>> > >> >> > tool.
>> > >> >> > > >
>> > >> >> > > > --don
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
So if I send a PR removing the mysql connector (which comes from maven
which is itself apache released), can we be done w/ this?
I don't want to get into the fact that it runs on linux and uses bash in
scripts.
I think there's some confusion in this thread about what people expect from
container images. We want this to be the standard image in the Kubernetes
helm chart as an exxample. No one wants to buiild their own container and
find their own registry to host it.
But if its solely about mysql, I think removing it is fine, the postgres
connector works at least as well.
On the 1/2/3 list:
#1: I'll do the PR to remove mysql
#2. I cannot do this, I have no idea what Jira or apache infra is etc.
#3. I don't think this is worth doing, no one will use it except for
someone who can trivially do that themself and might want other things in
there anyway.
so, if someone can help w/ #2 (getting the travis build to push this or
whatever) i'll do a PR for #1.
OK?
On Tue, 5 Mar 2019 at 12:59, Charles Allen <ch...@snap.com.invalid>
wrote:
> Honestly we're at a very strange impasse. On one hand I don't think the ASF
> project can adopt an official docker image unless ASF legal says its ok.
> "Official" releases are source code anyways (as my understanding goes), and
> binary artifacts are convenience things. Unfortunately I do not see a path
> forward unless some entity is willing to take on a stance similar as
> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> pretty new territory from a legal perspective (the fact that docker images
> are layers makes it even more interesting).
>
> At this point I think the safest thing to do is something that is "no more
> GPL dependent than other containers in the apache repo", which would mean
> not adding in GPL binaries. Which means switching to postgres. I don't
> foresee an aggressive legal stance on this issue, meaning it might take a
> while as people watch where the industry is going.
>
>
>
> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>
> > where do we stand on this?
> > the PR is in and accepted, but i feel we need to have this built as part
> of
> > the release artifacts and on dockerhub to foster adoption.
> > if the only issue is the mysql connector i can remove it in favour of the
> > postgres connector.
> >
> >
> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> >
> > > i can just remove the mysql, the postgres works, i was just assuming
> > folks
> > > wanted it.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> A discussion is progressing on
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> > It doesn't seem to have
> > >> got anywhere firm yet.
> > >>
> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > I don't think anything is strictly needed from you at this point,
> but
> > >> > things happen when people drive them, and participation in that
> effort
> > >> > would help make sure it gets done. I think at this point the tasks
> on
> > >> our
> > >> > end are watching LEGAL-437 for advice (or making it moot by removing
> > the
> > >> > MySQL jar), asking Infra to set up automated builds once that is
> > sorted
> > >> > out, and building some kind of consensus around how we'll label and
> > >> promote
> > >> > the Docker images.
> > >> >
> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> wrote:
> > >> >
> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > >> metadata.
> > >> >> if this is the case we should consider relfecting postgres as the
> > >> default
> > >> >> metadata in the docs.
> > >> >> however, i think this is mere aggregation under the gpl license,
> and
> > >> the
> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> start
> > >> >> scripts are all bash-specific as an example.
> > >> >>
> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> get
> > an
> > >> >> automated build going into dockerhub, and tagged w/ each release. i
> > >> think
> > >> >> this will help adoption.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > >> >>
> > >> >> > First off thanks a lot for your work here Don!!
> > >> >> >
> > >> >> > I really do think, though, that we need to be careful about the
> > >> >> inclusion
> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> > that
> > >> >> ASF
> > >> >> > projects should not distribute it as part of binary convenience
> > >> >> releases:
> > >> >> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> .
> > I think having the
> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
> > >> >> > distributing the jar itself, just, essentially, a pointer to how
> to
> > >> >> > download it. But if we start offering a prebuilt Docker image, it
> > is
> > >> >> less
> > >> >> > clear to me if that is fine or not. In the interests of resolving
> > >> this
> > >> >> > question one way or the other, I opened a question asking about
> > this
> > >> >> > specific situation:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > >> >> >
> > >> >> > About Dylan's questions: my feeling is that we should go ahead
> and
> > >> >> enable
> > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > language
> > >> >> > around what people should expect out of it. I don't think
> > >> >> 'experimental' is
> > >> >> > the right word, but we should be clear around exactly what
> contract
> > >> we
> > >> >> are
> > >> >> > adhering to. Is it something people can expect to be published
> with
> > >> each
> > >> >> > release? Is it something that we are going to build and test as
> > part
> > >> of
> > >> >> the
> > >> >> > release process, or are we going to publish it via automation
> > without
> > >> >> any
> > >> >> > testing? Is it something we expect people to use in production,
> or
> > >> >> > something we only expect people to use for evaluation? If it is
> > >> >> something
> > >> >> > we expect people to use in production, do we expect them to use
> it
> > in
> > >> >> any
> > >> >> > particular way? Will we be guaranteeing certain things (file
> > layout,
> > >> >> etc)
> > >> >> > that provide a stable interface for people to build derived
> images
> > >> from?
> > >> >> >
> > >> >> > The path of least resistance to answering these questions is to
> say
> > >> that
> > >> >> > the Docker image is provided in the hopes that it is useful, but
> > that
> > >> >> it is
> > >> >> > done via an automated build, without any pre-release testing, and
> > >> >> without
> > >> >> > any particular guarantees about the 'interface' it provides. If
> > this
> > >> is
> > >> >> the
> > >> >> > case then I would suggest putting it up on Docker Hub with an
> > >> >> appropriate
> > >> >> > disclaimer and not promoting it too much. (We might very well end
> > up
> > >> >> > pushing images every once in a while that don't work right, and
> it
> > >> would
> > >> >> > reflect poorly on the project to have those be prominently
> > >> linked-to.)
> > >> >> It
> > >> >> > becomes easier to strengthen these guarantees if we add an
> > automated
> > >> >> test
> > >> >> > suite that we can run before releases which verifies
> functionality
> > >> and
> > >> >> > interface adherence.
> > >> >> >
> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > >> >> <rm...@vmware.com.invalid>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > This is purely a packaging exercise. I don't see a reason to
> mark
> > >> >> this as
> > >> >> > > experimental.
> > >> >> > >
> > >> >> > > Rajiv.
> > >> >> > > ________________________________
> > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > >> >> > > To: dev@druid.apache.org
> > >> >> > > Subject: Re: docker build
> > >> >> > >
> > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > >> >> Infrastructure
> > >> >> > > team and then we'll have some automatic process that'll
> > >> automatically
> > >> >> > > update docker-hub with images relating to each release.
> > >> >> > >
> > >> >> > > I guess there's two open questions I think we should reach a
> > >> >> consensus on
> > >> >> > > (others feel free to add more!).
> > >> >> > >
> > >> >> > > - Are we as a community happy to "support" an additional
> release
> > >> >> > artefact?
> > >> >> > > I'm happy to try to incorporate this into my employer's testing
> > >> >> > > infrastructure to help catch any regressions on future releases
> > but
> > >> >> > that's
> > >> >> > > just one data point on each release.
> > >> >> > >
> > >> >> > > - Along the same vein, do we follow the same process as we do
> > with
> > >> new
> > >> >> > > features and mark this as experimental for some time?
> > >> >> > >
> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > wrote:
> > >> >> > >
> > >> >> > > > Now that
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > >> >> > > is merged
> > >> >> > > > (thank you!)
> > >> >> > > >
> > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > >> >> automatically
> > >> >> > > on a
> > >> >> > > > 'tag' of the repo.
> > >> >> > > >
> > >> >> > > > Once that is done it is much more convenient for folks to use
> > >> this
> > >> >> > tool.
> > >> >> > > >
> > >> >> > > > --don
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
Honestly we're at a very strange impasse. On one hand I don't think the ASF
project can adopt an official docker image unless ASF legal says its ok.
"Official" releases are source code anyways (as my understanding goes), and
binary artifacts are convenience things. Unfortunately I do not see a path
forward unless some entity is willing to take on a stance similar as
outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
pretty new territory from a legal perspective (the fact that docker images
are layers makes it even more interesting).
At this point I think the safest thing to do is something that is "no more
GPL dependent than other containers in the apache repo", which would mean
not adding in GPL binaries. Which means switching to postgres. I don't
foresee an aggressive legal stance on this issue, meaning it might take a
while as people watch where the industry is going.
On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> where do we stand on this?
> the PR is in and accepted, but i feel we need to have this built as part of
> the release artifacts and on dockerhub to foster adoption.
> if the only issue is the mysql connector i can remove it in favour of the
> postgres connector.
>
>
> On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
>
> > i can just remove the mysql, the postgres works, i was just assuming
> folks
> > wanted it.
> >
> >
> > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> >
> >> A discussion is progressing on
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=.
> It doesn't seem to have
> >> got anywhere firm yet.
> >>
> >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > I don't think anything is strictly needed from you at this point, but
> >> > things happen when people drive them, and participation in that effort
> >> > would help make sure it gets done. I think at this point the tasks on
> >> our
> >> > end are watching LEGAL-437 for advice (or making it moot by removing
> the
> >> > MySQL jar), asking Infra to set up automated builds once that is
> sorted
> >> > out, and building some kind of consensus around how we'll label and
> >> promote
> >> > the Docker images.
> >> >
> >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> >> >
> >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> >> metadata.
> >> >> if this is the case we should consider relfecting postgres as the
> >> default
> >> >> metadata in the docs.
> >> >> however, i think this is mere aggregation under the gpl license, and
> >> the
> >> >> docker image tends to have other (e.g. bash) gpl code. druid's start
> >> >> scripts are all bash-specific as an example.
> >> >>
> >> >> I'm not clear if anything further is needed of me, i'm hoping to get
> an
> >> >> automated build going into dockerhub, and tagged w/ each release. i
> >> think
> >> >> this will help adoption.
> >> >>
> >> >>
> >> >>
> >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> >> >>
> >> >> > First off thanks a lot for your work here Don!!
> >> >> >
> >> >> > I really do think, though, that we need to be careful about the
> >> >> inclusion
> >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> that
> >> >> ASF
> >> >> > projects should not distribute it as part of binary convenience
> >> >> releases:
> >> >> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=.
> I think having the
> >> >> > Dockerfile in the repo is probably fine: in that case we are not
> >> >> > distributing the jar itself, just, essentially, a pointer to how to
> >> >> > download it. But if we start offering a prebuilt Docker image, it
> is
> >> >> less
> >> >> > clear to me if that is fine or not. In the interests of resolving
> >> this
> >> >> > question one way or the other, I opened a question asking about
> this
> >> >> > specific situation:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> >> >> >
> >> >> > About Dylan's questions: my feeling is that we should go ahead and
> >> >> enable
> >> >> > automated pushes to Docker Hub, and provide some appropriate
> language
> >> >> > around what people should expect out of it. I don't think
> >> >> 'experimental' is
> >> >> > the right word, but we should be clear around exactly what contract
> >> we
> >> >> are
> >> >> > adhering to. Is it something people can expect to be published with
> >> each
> >> >> > release? Is it something that we are going to build and test as
> part
> >> of
> >> >> the
> >> >> > release process, or are we going to publish it via automation
> without
> >> >> any
> >> >> > testing? Is it something we expect people to use in production, or
> >> >> > something we only expect people to use for evaluation? If it is
> >> >> something
> >> >> > we expect people to use in production, do we expect them to use it
> in
> >> >> any
> >> >> > particular way? Will we be guaranteeing certain things (file
> layout,
> >> >> etc)
> >> >> > that provide a stable interface for people to build derived images
> >> from?
> >> >> >
> >> >> > The path of least resistance to answering these questions is to say
> >> that
> >> >> > the Docker image is provided in the hopes that it is useful, but
> that
> >> >> it is
> >> >> > done via an automated build, without any pre-release testing, and
> >> >> without
> >> >> > any particular guarantees about the 'interface' it provides. If
> this
> >> is
> >> >> the
> >> >> > case then I would suggest putting it up on Docker Hub with an
> >> >> appropriate
> >> >> > disclaimer and not promoting it too much. (We might very well end
> up
> >> >> > pushing images every once in a while that don't work right, and it
> >> would
> >> >> > reflect poorly on the project to have those be prominently
> >> linked-to.)
> >> >> It
> >> >> > becomes easier to strengthen these guarantees if we add an
> automated
> >> >> test
> >> >> > suite that we can run before releases which verifies functionality
> >> and
> >> >> > interface adherence.
> >> >> >
> >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> >> >> <rm...@vmware.com.invalid>
> >> >> > wrote:
> >> >> >
> >> >> > > This is purely a packaging exercise. I don't see a reason to mark
> >> >> this as
> >> >> > > experimental.
> >> >> > >
> >> >> > > Rajiv.
> >> >> > > ________________________________
> >> >> > > From: Dylan Wylie <dy...@apache.org>
> >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> >> >> > > To: dev@druid.apache.org
> >> >> > > Subject: Re: docker build
> >> >> > >
> >> >> > > I believe all we have to do is submit a ticket to Apache's
> >> >> Infrastructure
> >> >> > > team and then we'll have some automatic process that'll
> >> automatically
> >> >> > > update docker-hub with images relating to each release.
> >> >> > >
> >> >> > > I guess there's two open questions I think we should reach a
> >> >> consensus on
> >> >> > > (others feel free to add more!).
> >> >> > >
> >> >> > > - Are we as a community happy to "support" an additional release
> >> >> > artefact?
> >> >> > > I'm happy to try to incorporate this into my employer's testing
> >> >> > > infrastructure to help catch any regressions on future releases
> but
> >> >> > that's
> >> >> > > just one data point on each release.
> >> >> > >
> >> >> > > - Along the same vein, do we follow the same process as we do
> with
> >> new
> >> >> > > features and mark this as experimental for some time?
> >> >> > >
> >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> wrote:
> >> >> > >
> >> >> > > > Now that
> >> >> > >
> >> >> >
> >> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> >> >> > > is merged
> >> >> > > > (thank you!)
> >> >> > > >
> >> >> > > > who can get this set to build into Dockerhub? Presumably
> >> >> automatically
> >> >> > > on a
> >> >> > > > 'tag' of the repo.
> >> >> > > >
> >> >> > > > Once that is done it is much more convenient for folks to use
> >> this
> >> >> > tool.
> >> >> > > >
> >> >> > > > --don
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>