You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Don Bowman <do...@agilicus.com> on 2019/02/08 13:25:17 UTC
docker build
Now that https://github.com/apache/incubator-druid/pull/6896 is merged
(thank you!)
who can get this set to build into Dockerhub? Presumably automatically on a
'tag' of the repo.
Once that is done it is much more convenient for folks to use this tool.
--don
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
David, that plan looks good to me too. Is anyone interested in implementing
it (maybe David or Don?). Seems to me like the steps needed would be,
1) Do a PR to implement the Dockerfile changes in the Druid repo (remove
MySQL from the Dockerfile that will be used for DockerHub).
2) Work with Infra to set up automated tag builds. Examples -
https://jira.apache.org/jira/browse/INFRA-13838,
https://jira.apache.org/jira/browse/INFRA-12781.
3) Potentially add another Dockerfile FROM the autobuilt image that adds
MySQL and that interested users could run on their own.
On Tue, Mar 5, 2019 at 11:19 AM Charles Allen
<ch...@snap.com.invalid> wrote:
> I would support that.
>
> On Tue, Mar 5, 2019 at 11:13 AM David Glasser <gl...@apollographql.com>
> wrote:
>
> > Is the following a reasonable solution from both a usability and
> > legal perspective?
> > - Write a Dockerfile that has everything except the GPL jar on it
> > (including the Druid code that talks to the jar if configured to use
> MySQL)
> > - Automatically publish that Docker image to the ASF account on DockerHub
> > - Also include a short Dockerfile in the repo that starts `FROM` our
> > auto-built account and has a line or two of wget to download the jar
> > (similar to the wget currently in the Dockerfile)
> > - Tell users who want to use MySQL that they must publish that extra
> > layered image themselves
> >
> > --dave
> >
> >
> > On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <charles.allen@snap.com
> > .invalid>
> > wrote:
> >
> > > Honestly we're at a very strange impasse. On one hand I don't think the
> > ASF
> > > project can adopt an official docker image unless ASF legal says its
> ok.
> > > "Official" releases are source code anyways (as my understanding goes),
> > and
> > > binary artifacts are convenience things. Unfortunately I do not see a
> > path
> > > forward unless some entity is willing to take on a stance similar as
> > > outlined in
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=x7yglOWgfT8oqHqcgn_BnihcsuEZ-gE5QzWksV1PCDg&s=XPfeziOkU4SCezu5EyBUj-pEGcxOY7XVHU1rdNobwrw&e=
> > . This is
> > > pretty new territory from a legal perspective (the fact that docker
> > images
> > > are layers makes it even more interesting).
> > >
> > > At this point I think the safest thing to do is something that is "no
> > more
> > > GPL dependent than other containers in the apache repo", which would
> mean
> > > not adding in GPL binaries. Which means switching to postgres. I don't
> > > foresee an aggressive legal stance on this issue, meaning it might
> take a
> > > while as people watch where the industry is going.
> > >
> > >
> > >
> > > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > where do we stand on this?
> > > > the PR is in and accepted, but i feel we need to have this built as
> > part
> > > of
> > > > the release artifacts and on dockerhub to foster adoption.
> > > > if the only issue is the mysql connector i can remove it in favour of
> > the
> > > > postgres connector.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > > >
> > > > > i can just remove the mysql, the postgres works, i was just
> assuming
> > > > folks
> > > > > wanted it.
> > > > >
> > > > >
> > > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org>
> wrote:
> > > > >
> > > > >> A discussion is progressing on
> > > > >>
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > > It doesn't seem to have
> > > > >> got anywhere firm yet.
> > > > >>
> > > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >>
> > > > >> > I don't think anything is strictly needed from you at this
> point,
> > > but
> > > > >> > things happen when people drive them, and participation in that
> > > effort
> > > > >> > would help make sure it gets done. I think at this point the
> tasks
> > > on
> > > > >> our
> > > > >> > end are watching LEGAL-437 for advice (or making it moot by
> > removing
> > > > the
> > > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > > sorted
> > > > >> > out, and building some kind of consensus around how we'll label
> > and
> > > > >> promote
> > > > >> > the Docker images.
> > > > >> >
> > > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > > wrote:
> > > > >> >
> > > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > > >> metadata.
> > > > >> >> if this is the case we should consider relfecting postgres as
> the
> > > > >> default
> > > > >> >> metadata in the docs.
> > > > >> >> however, i think this is mere aggregation under the gpl
> license,
> > > and
> > > > >> the
> > > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > > start
> > > > >> >> scripts are all bash-specific as an example.
> > > > >> >>
> > > > >> >> I'm not clear if anything further is needed of me, i'm hoping
> to
> > > get
> > > > an
> > > > >> >> automated build going into dockerhub, and tagged w/ each
> > release. i
> > > > >> think
> > > > >> >> this will help adoption.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >> >>
> > > > >> >> > First off thanks a lot for your work here Don!!
> > > > >> >> >
> > > > >> >> > I really do think, though, that we need to be careful about
> the
> > > > >> >> inclusion
> > > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> > past
> > > > that
> > > > >> >> ASF
> > > > >> >> > projects should not distribute it as part of binary
> convenience
> > > > >> >> releases:
> > > > >> >> >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > > .
> > > > I think having the
> > > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> > not
> > > > >> >> > distributing the jar itself, just, essentially, a pointer to
> > how
> > > to
> > > > >> >> > download it. But if we start offering a prebuilt Docker
> image,
> > it
> > > > is
> > > > >> >> less
> > > > >> >> > clear to me if that is fine or not. In the interests of
> > resolving
> > > > >> this
> > > > >> >> > question one way or the other, I opened a question asking
> about
> > > > this
> > > > >> >> > specific situation:
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > > .
> > > > >> >> >
> > > > >> >> > About Dylan's questions: my feeling is that we should go
> ahead
> > > and
> > > > >> >> enable
> > > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > > language
> > > > >> >> > around what people should expect out of it. I don't think
> > > > >> >> 'experimental' is
> > > > >> >> > the right word, but we should be clear around exactly what
> > > contract
> > > > >> we
> > > > >> >> are
> > > > >> >> > adhering to. Is it something people can expect to be
> published
> > > with
> > > > >> each
> > > > >> >> > release? Is it something that we are going to build and test
> as
> > > > part
> > > > >> of
> > > > >> >> the
> > > > >> >> > release process, or are we going to publish it via automation
> > > > without
> > > > >> >> any
> > > > >> >> > testing? Is it something we expect people to use in
> production,
> > > or
> > > > >> >> > something we only expect people to use for evaluation? If it
> is
> > > > >> >> something
> > > > >> >> > we expect people to use in production, do we expect them to
> use
> > > it
> > > > in
> > > > >> >> any
> > > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > > layout,
> > > > >> >> etc)
> > > > >> >> > that provide a stable interface for people to build derived
> > > images
> > > > >> from?
> > > > >> >> >
> > > > >> >> > The path of least resistance to answering these questions is
> to
> > > say
> > > > >> that
> > > > >> >> > the Docker image is provided in the hopes that it is useful,
> > but
> > > > that
> > > > >> >> it is
> > > > >> >> > done via an automated build, without any pre-release testing,
> > and
> > > > >> >> without
> > > > >> >> > any particular guarantees about the 'interface' it provides.
> If
> > > > this
> > > > >> is
> > > > >> >> the
> > > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > > >> >> appropriate
> > > > >> >> > disclaimer and not promoting it too much. (We might very well
> > end
> > > > up
> > > > >> >> > pushing images every once in a while that don't work right,
> and
> > > it
> > > > >> would
> > > > >> >> > reflect poorly on the project to have those be prominently
> > > > >> linked-to.)
> > > > >> >> It
> > > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > > automated
> > > > >> >> test
> > > > >> >> > suite that we can run before releases which verifies
> > > functionality
> > > > >> and
> > > > >> >> > interface adherence.
> > > > >> >> >
> > > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > > >> >> <rm...@vmware.com.invalid>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> > > This is purely a packaging exercise. I don't see a reason
> to
> > > mark
> > > > >> >> this as
> > > > >> >> > > experimental.
> > > > >> >> > >
> > > > >> >> > > Rajiv.
> > > > >> >> > > ________________________________
> > > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > > >> >> > > To: dev@druid.apache.org
> > > > >> >> > > Subject: Re: docker build
> > > > >> >> > >
> > > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > > >> >> Infrastructure
> > > > >> >> > > team and then we'll have some automatic process that'll
> > > > >> automatically
> > > > >> >> > > update docker-hub with images relating to each release.
> > > > >> >> > >
> > > > >> >> > > I guess there's two open questions I think we should reach
> a
> > > > >> >> consensus on
> > > > >> >> > > (others feel free to add more!).
> > > > >> >> > >
> > > > >> >> > > - Are we as a community happy to "support" an additional
> > > release
> > > > >> >> > artefact?
> > > > >> >> > > I'm happy to try to incorporate this into my employer's
> > testing
> > > > >> >> > > infrastructure to help catch any regressions on future
> > releases
> > > > but
> > > > >> >> > that's
> > > > >> >> > > just one data point on each release.
> > > > >> >> > >
> > > > >> >> > > - Along the same vein, do we follow the same process as we
> do
> > > > with
> > > > >> new
> > > > >> >> > > features and mark this as experimental for some time?
> > > > >> >> > >
> > > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > > wrote:
> > > > >> >> > >
> > > > >> >> > > > Now that
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > > >> >> > > is merged
> > > > >> >> > > > (thank you!)
> > > > >> >> > > >
> > > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > > >> >> automatically
> > > > >> >> > > on a
> > > > >> >> > > > 'tag' of the repo.
> > > > >> >> > > >
> > > > >> >> > > > Once that is done it is much more convenient for folks to
> > use
> > > > >> this
> > > > >> >> > tool.
> > > > >> >> > > >
> > > > >> >> > > > --don
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
I would support that.
On Tue, Mar 5, 2019 at 11:13 AM David Glasser <gl...@apollographql.com>
wrote:
> Is the following a reasonable solution from both a usability and
> legal perspective?
> - Write a Dockerfile that has everything except the GPL jar on it
> (including the Druid code that talks to the jar if configured to use MySQL)
> - Automatically publish that Docker image to the ASF account on DockerHub
> - Also include a short Dockerfile in the repo that starts `FROM` our
> auto-built account and has a line or two of wget to download the jar
> (similar to the wget currently in the Dockerfile)
> - Tell users who want to use MySQL that they must publish that extra
> layered image themselves
>
> --dave
>
>
> On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <charles.allen@snap.com
> .invalid>
> wrote:
>
> > Honestly we're at a very strange impasse. On one hand I don't think the
> ASF
> > project can adopt an official docker image unless ASF legal says its ok.
> > "Official" releases are source code anyways (as my understanding goes),
> and
> > binary artifacts are convenience things. Unfortunately I do not see a
> path
> > forward unless some entity is willing to take on a stance similar as
> > outlined in
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=x7yglOWgfT8oqHqcgn_BnihcsuEZ-gE5QzWksV1PCDg&s=XPfeziOkU4SCezu5EyBUj-pEGcxOY7XVHU1rdNobwrw&e=
> . This is
> > pretty new territory from a legal perspective (the fact that docker
> images
> > are layers makes it even more interesting).
> >
> > At this point I think the safest thing to do is something that is "no
> more
> > GPL dependent than other containers in the apache repo", which would mean
> > not adding in GPL binaries. Which means switching to postgres. I don't
> > foresee an aggressive legal stance on this issue, meaning it might take a
> > while as people watch where the industry is going.
> >
> >
> >
> > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> >
> > > where do we stand on this?
> > > the PR is in and accepted, but i feel we need to have this built as
> part
> > of
> > > the release artifacts and on dockerhub to foster adoption.
> > > if the only issue is the mysql connector i can remove it in favour of
> the
> > > postgres connector.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > i can just remove the mysql, the postgres works, i was just assuming
> > > folks
> > > > wanted it.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > >> A discussion is progressing on
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > > It doesn't seem to have
> > > >> got anywhere firm yet.
> > > >>
> > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >>
> > > >> > I don't think anything is strictly needed from you at this point,
> > but
> > > >> > things happen when people drive them, and participation in that
> > effort
> > > >> > would help make sure it gets done. I think at this point the tasks
> > on
> > > >> our
> > > >> > end are watching LEGAL-437 for advice (or making it moot by
> removing
> > > the
> > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > sorted
> > > >> > out, and building some kind of consensus around how we'll label
> and
> > > >> promote
> > > >> > the Docker images.
> > > >> >
> > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > wrote:
> > > >> >
> > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > >> metadata.
> > > >> >> if this is the case we should consider relfecting postgres as the
> > > >> default
> > > >> >> metadata in the docs.
> > > >> >> however, i think this is mere aggregation under the gpl license,
> > and
> > > >> the
> > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > start
> > > >> >> scripts are all bash-specific as an example.
> > > >> >>
> > > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> > get
> > > an
> > > >> >> automated build going into dockerhub, and tagged w/ each
> release. i
> > > >> think
> > > >> >> this will help adoption.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> wrote:
> > > >> >>
> > > >> >> > First off thanks a lot for your work here Don!!
> > > >> >> >
> > > >> >> > I really do think, though, that we need to be careful about the
> > > >> >> inclusion
> > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> past
> > > that
> > > >> >> ASF
> > > >> >> > projects should not distribute it as part of binary convenience
> > > >> >> releases:
> > > >> >> >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > .
> > > I think having the
> > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> not
> > > >> >> > distributing the jar itself, just, essentially, a pointer to
> how
> > to
> > > >> >> > download it. But if we start offering a prebuilt Docker image,
> it
> > > is
> > > >> >> less
> > > >> >> > clear to me if that is fine or not. In the interests of
> resolving
> > > >> this
> > > >> >> > question one way or the other, I opened a question asking about
> > > this
> > > >> >> > specific situation:
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > >> >> >
> > > >> >> > About Dylan's questions: my feeling is that we should go ahead
> > and
> > > >> >> enable
> > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > language
> > > >> >> > around what people should expect out of it. I don't think
> > > >> >> 'experimental' is
> > > >> >> > the right word, but we should be clear around exactly what
> > contract
> > > >> we
> > > >> >> are
> > > >> >> > adhering to. Is it something people can expect to be published
> > with
> > > >> each
> > > >> >> > release? Is it something that we are going to build and test as
> > > part
> > > >> of
> > > >> >> the
> > > >> >> > release process, or are we going to publish it via automation
> > > without
> > > >> >> any
> > > >> >> > testing? Is it something we expect people to use in production,
> > or
> > > >> >> > something we only expect people to use for evaluation? If it is
> > > >> >> something
> > > >> >> > we expect people to use in production, do we expect them to use
> > it
> > > in
> > > >> >> any
> > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > layout,
> > > >> >> etc)
> > > >> >> > that provide a stable interface for people to build derived
> > images
> > > >> from?
> > > >> >> >
> > > >> >> > The path of least resistance to answering these questions is to
> > say
> > > >> that
> > > >> >> > the Docker image is provided in the hopes that it is useful,
> but
> > > that
> > > >> >> it is
> > > >> >> > done via an automated build, without any pre-release testing,
> and
> > > >> >> without
> > > >> >> > any particular guarantees about the 'interface' it provides. If
> > > this
> > > >> is
> > > >> >> the
> > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > >> >> appropriate
> > > >> >> > disclaimer and not promoting it too much. (We might very well
> end
> > > up
> > > >> >> > pushing images every once in a while that don't work right, and
> > it
> > > >> would
> > > >> >> > reflect poorly on the project to have those be prominently
> > > >> linked-to.)
> > > >> >> It
> > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > automated
> > > >> >> test
> > > >> >> > suite that we can run before releases which verifies
> > functionality
> > > >> and
> > > >> >> > interface adherence.
> > > >> >> >
> > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > >> >> <rm...@vmware.com.invalid>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > This is purely a packaging exercise. I don't see a reason to
> > mark
> > > >> >> this as
> > > >> >> > > experimental.
> > > >> >> > >
> > > >> >> > > Rajiv.
> > > >> >> > > ________________________________
> > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > >> >> > > To: dev@druid.apache.org
> > > >> >> > > Subject: Re: docker build
> > > >> >> > >
> > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > >> >> Infrastructure
> > > >> >> > > team and then we'll have some automatic process that'll
> > > >> automatically
> > > >> >> > > update docker-hub with images relating to each release.
> > > >> >> > >
> > > >> >> > > I guess there's two open questions I think we should reach a
> > > >> >> consensus on
> > > >> >> > > (others feel free to add more!).
> > > >> >> > >
> > > >> >> > > - Are we as a community happy to "support" an additional
> > release
> > > >> >> > artefact?
> > > >> >> > > I'm happy to try to incorporate this into my employer's
> testing
> > > >> >> > > infrastructure to help catch any regressions on future
> releases
> > > but
> > > >> >> > that's
> > > >> >> > > just one data point on each release.
> > > >> >> > >
> > > >> >> > > - Along the same vein, do we follow the same process as we do
> > > with
> > > >> new
> > > >> >> > > features and mark this as experimental for some time?
> > > >> >> > >
> > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > wrote:
> > > >> >> > >
> > > >> >> > > > Now that
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > >> >> > > is merged
> > > >> >> > > > (thank you!)
> > > >> >> > > >
> > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > >> >> automatically
> > > >> >> > > on a
> > > >> >> > > > 'tag' of the repo.
> > > >> >> > > >
> > > >> >> > > > Once that is done it is much more convenient for folks to
> use
> > > >> this
> > > >> >> > tool.
> > > >> >> > > >
> > > >> >> > > > --don
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: docker build
Posted by David Glasser <gl...@apollographql.com>.
Is the following a reasonable solution from both a usability and
legal perspective?
- Write a Dockerfile that has everything except the GPL jar on it
(including the Druid code that talks to the jar if configured to use MySQL)
- Automatically publish that Docker image to the ASF account on DockerHub
- Also include a short Dockerfile in the repo that starts `FROM` our
auto-built account and has a line or two of wget to download the jar
(similar to the wget currently in the Dockerfile)
- Tell users who want to use MySQL that they must publish that extra
layered image themselves
--dave
On Tue, Mar 5, 2019 at 9:59 AM Charles Allen <ch...@snap.com.invalid>
wrote:
> Honestly we're at a very strange impasse. On one hand I don't think the ASF
> project can adopt an official docker image unless ASF legal says its ok.
> "Official" releases are source code anyways (as my understanding goes), and
> binary artifacts are convenience things. Unfortunately I do not see a path
> forward unless some entity is willing to take on a stance similar as
> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> pretty new territory from a legal perspective (the fact that docker images
> are layers makes it even more interesting).
>
> At this point I think the safest thing to do is something that is "no more
> GPL dependent than other containers in the apache repo", which would mean
> not adding in GPL binaries. Which means switching to postgres. I don't
> foresee an aggressive legal stance on this issue, meaning it might take a
> while as people watch where the industry is going.
>
>
>
> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>
> > where do we stand on this?
> > the PR is in and accepted, but i feel we need to have this built as part
> of
> > the release artifacts and on dockerhub to foster adoption.
> > if the only issue is the mysql connector i can remove it in favour of the
> > postgres connector.
> >
> >
> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> >
> > > i can just remove the mysql, the postgres works, i was just assuming
> > folks
> > > wanted it.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> A discussion is progressing on
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> > It doesn't seem to have
> > >> got anywhere firm yet.
> > >>
> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > I don't think anything is strictly needed from you at this point,
> but
> > >> > things happen when people drive them, and participation in that
> effort
> > >> > would help make sure it gets done. I think at this point the tasks
> on
> > >> our
> > >> > end are watching LEGAL-437 for advice (or making it moot by removing
> > the
> > >> > MySQL jar), asking Infra to set up automated builds once that is
> > sorted
> > >> > out, and building some kind of consensus around how we'll label and
> > >> promote
> > >> > the Docker images.
> > >> >
> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> wrote:
> > >> >
> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > >> metadata.
> > >> >> if this is the case we should consider relfecting postgres as the
> > >> default
> > >> >> metadata in the docs.
> > >> >> however, i think this is mere aggregation under the gpl license,
> and
> > >> the
> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> start
> > >> >> scripts are all bash-specific as an example.
> > >> >>
> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> get
> > an
> > >> >> automated build going into dockerhub, and tagged w/ each release. i
> > >> think
> > >> >> this will help adoption.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > >> >>
> > >> >> > First off thanks a lot for your work here Don!!
> > >> >> >
> > >> >> > I really do think, though, that we need to be careful about the
> > >> >> inclusion
> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> > that
> > >> >> ASF
> > >> >> > projects should not distribute it as part of binary convenience
> > >> >> releases:
> > >> >> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> .
> > I think having the
> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
> > >> >> > distributing the jar itself, just, essentially, a pointer to how
> to
> > >> >> > download it. But if we start offering a prebuilt Docker image, it
> > is
> > >> >> less
> > >> >> > clear to me if that is fine or not. In the interests of resolving
> > >> this
> > >> >> > question one way or the other, I opened a question asking about
> > this
> > >> >> > specific situation:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > >> >> >
> > >> >> > About Dylan's questions: my feeling is that we should go ahead
> and
> > >> >> enable
> > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > language
> > >> >> > around what people should expect out of it. I don't think
> > >> >> 'experimental' is
> > >> >> > the right word, but we should be clear around exactly what
> contract
> > >> we
> > >> >> are
> > >> >> > adhering to. Is it something people can expect to be published
> with
> > >> each
> > >> >> > release? Is it something that we are going to build and test as
> > part
> > >> of
> > >> >> the
> > >> >> > release process, or are we going to publish it via automation
> > without
> > >> >> any
> > >> >> > testing? Is it something we expect people to use in production,
> or
> > >> >> > something we only expect people to use for evaluation? If it is
> > >> >> something
> > >> >> > we expect people to use in production, do we expect them to use
> it
> > in
> > >> >> any
> > >> >> > particular way? Will we be guaranteeing certain things (file
> > layout,
> > >> >> etc)
> > >> >> > that provide a stable interface for people to build derived
> images
> > >> from?
> > >> >> >
> > >> >> > The path of least resistance to answering these questions is to
> say
> > >> that
> > >> >> > the Docker image is provided in the hopes that it is useful, but
> > that
> > >> >> it is
> > >> >> > done via an automated build, without any pre-release testing, and
> > >> >> without
> > >> >> > any particular guarantees about the 'interface' it provides. If
> > this
> > >> is
> > >> >> the
> > >> >> > case then I would suggest putting it up on Docker Hub with an
> > >> >> appropriate
> > >> >> > disclaimer and not promoting it too much. (We might very well end
> > up
> > >> >> > pushing images every once in a while that don't work right, and
> it
> > >> would
> > >> >> > reflect poorly on the project to have those be prominently
> > >> linked-to.)
> > >> >> It
> > >> >> > becomes easier to strengthen these guarantees if we add an
> > automated
> > >> >> test
> > >> >> > suite that we can run before releases which verifies
> functionality
> > >> and
> > >> >> > interface adherence.
> > >> >> >
> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > >> >> <rm...@vmware.com.invalid>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > This is purely a packaging exercise. I don't see a reason to
> mark
> > >> >> this as
> > >> >> > > experimental.
> > >> >> > >
> > >> >> > > Rajiv.
> > >> >> > > ________________________________
> > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > >> >> > > To: dev@druid.apache.org
> > >> >> > > Subject: Re: docker build
> > >> >> > >
> > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > >> >> Infrastructure
> > >> >> > > team and then we'll have some automatic process that'll
> > >> automatically
> > >> >> > > update docker-hub with images relating to each release.
> > >> >> > >
> > >> >> > > I guess there's two open questions I think we should reach a
> > >> >> consensus on
> > >> >> > > (others feel free to add more!).
> > >> >> > >
> > >> >> > > - Are we as a community happy to "support" an additional
> release
> > >> >> > artefact?
> > >> >> > > I'm happy to try to incorporate this into my employer's testing
> > >> >> > > infrastructure to help catch any regressions on future releases
> > but
> > >> >> > that's
> > >> >> > > just one data point on each release.
> > >> >> > >
> > >> >> > > - Along the same vein, do we follow the same process as we do
> > with
> > >> new
> > >> >> > > features and mark this as experimental for some time?
> > >> >> > >
> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > wrote:
> > >> >> > >
> > >> >> > > > Now that
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > >> >> > > is merged
> > >> >> > > > (thank you!)
> > >> >> > > >
> > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > >> >> automatically
> > >> >> > > on a
> > >> >> > > > 'tag' of the repo.
> > >> >> > > >
> > >> >> > > > Once that is done it is much more convenient for folks to use
> > >> this
> > >> >> > tool.
> > >> >> > > >
> > >> >> > > > --don
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
Hey Don,
If you are interested in participating more directly in the Druid project,
you could certainly do #2. Apache projects are built around the idea of
community contribution, and people driving & building consensus on the
changes they want to see happen. Apache Infra is a group of people that
manage the Apache infrastructure: https://www.apache.org/dev/infrastructure.
They set up our mailing lists, git repos, etc, and can also set up
DockerHub builds. They use JIRA to track requests. Anyway, it is up to you.
On Tue, Mar 19, 2019 at 4:19 AM Don Bowman <do...@agilicus.com> wrote:
> So if I send a PR removing the mysql connector (which comes from maven
> which is itself apache released), can we be done w/ this?
>
> I don't want to get into the fact that it runs on linux and uses bash in
> scripts.
>
> I think there's some confusion in this thread about what people expect from
> container images. We want this to be the standard image in the Kubernetes
> helm chart as an exxample. No one wants to buiild their own container and
> find their own registry to host it.
>
> But if its solely about mysql, I think removing it is fine, the postgres
> connector works at least as well.
>
> On the 1/2/3 list:
>
> #1: I'll do the PR to remove mysql
> #2. I cannot do this, I have no idea what Jira or apache infra is etc.
> #3. I don't think this is worth doing, no one will use it except for
> someone who can trivially do that themself and might want other things in
> there anyway.
>
> so, if someone can help w/ #2 (getting the travis build to push this or
> whatever) i'll do a PR for #1.
> OK?
>
> On Tue, 5 Mar 2019 at 12:59, Charles Allen <charles.allen@snap.com
> .invalid>
> wrote:
>
> > Honestly we're at a very strange impasse. On one hand I don't think the
> ASF
> > project can adopt an official docker image unless ASF legal says its ok.
> > "Official" releases are source code anyways (as my understanding goes),
> and
> > binary artifacts are convenience things. Unfortunately I do not see a
> path
> > forward unless some entity is willing to take on a stance similar as
> > outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> > pretty new territory from a legal perspective (the fact that docker
> images
> > are layers makes it even more interesting).
> >
> > At this point I think the safest thing to do is something that is "no
> more
> > GPL dependent than other containers in the apache repo", which would mean
> > not adding in GPL binaries. Which means switching to postgres. I don't
> > foresee an aggressive legal stance on this issue, meaning it might take a
> > while as people watch where the industry is going.
> >
> >
> >
> > On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> >
> > > where do we stand on this?
> > > the PR is in and accepted, but i feel we need to have this built as
> part
> > of
> > > the release artifacts and on dockerhub to foster adoption.
> > > if the only issue is the mysql connector i can remove it in favour of
> the
> > > postgres connector.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > i can just remove the mysql, the postgres works, i was just assuming
> > > folks
> > > > wanted it.
> > > >
> > > >
> > > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > >> A discussion is progressing on
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > > It doesn't seem to have
> > > >> got anywhere firm yet.
> > > >>
> > > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >>
> > > >> > I don't think anything is strictly needed from you at this point,
> > but
> > > >> > things happen when people drive them, and participation in that
> > effort
> > > >> > would help make sure it gets done. I think at this point the tasks
> > on
> > > >> our
> > > >> > end are watching LEGAL-437 for advice (or making it moot by
> removing
> > > the
> > > >> > MySQL jar), asking Infra to set up automated builds once that is
> > > sorted
> > > >> > out, and building some kind of consensus around how we'll label
> and
> > > >> promote
> > > >> > the Docker images.
> > > >> >
> > > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> > wrote:
> > > >> >
> > > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > >> metadata.
> > > >> >> if this is the case we should consider relfecting postgres as the
> > > >> default
> > > >> >> metadata in the docs.
> > > >> >> however, i think this is mere aggregation under the gpl license,
> > and
> > > >> the
> > > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> > start
> > > >> >> scripts are all bash-specific as an example.
> > > >> >>
> > > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> > get
> > > an
> > > >> >> automated build going into dockerhub, and tagged w/ each
> release. i
> > > >> think
> > > >> >> this will help adoption.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> wrote:
> > > >> >>
> > > >> >> > First off thanks a lot for your work here Don!!
> > > >> >> >
> > > >> >> > I really do think, though, that we need to be careful about the
> > > >> >> inclusion
> > > >> >> > of the MySQL connector jar. ASF legal has been clear in the
> past
> > > that
> > > >> >> ASF
> > > >> >> > projects should not distribute it as part of binary convenience
> > > >> >> releases:
> > > >> >> >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> > .
> > > I think having the
> > > >> >> > Dockerfile in the repo is probably fine: in that case we are
> not
> > > >> >> > distributing the jar itself, just, essentially, a pointer to
> how
> > to
> > > >> >> > download it. But if we start offering a prebuilt Docker image,
> it
> > > is
> > > >> >> less
> > > >> >> > clear to me if that is fine or not. In the interests of
> resolving
> > > >> this
> > > >> >> > question one way or the other, I opened a question asking about
> > > this
> > > >> >> > specific situation:
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > > .
> > > >> >> >
> > > >> >> > About Dylan's questions: my feeling is that we should go ahead
> > and
> > > >> >> enable
> > > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > > language
> > > >> >> > around what people should expect out of it. I don't think
> > > >> >> 'experimental' is
> > > >> >> > the right word, but we should be clear around exactly what
> > contract
> > > >> we
> > > >> >> are
> > > >> >> > adhering to. Is it something people can expect to be published
> > with
> > > >> each
> > > >> >> > release? Is it something that we are going to build and test as
> > > part
> > > >> of
> > > >> >> the
> > > >> >> > release process, or are we going to publish it via automation
> > > without
> > > >> >> any
> > > >> >> > testing? Is it something we expect people to use in production,
> > or
> > > >> >> > something we only expect people to use for evaluation? If it is
> > > >> >> something
> > > >> >> > we expect people to use in production, do we expect them to use
> > it
> > > in
> > > >> >> any
> > > >> >> > particular way? Will we be guaranteeing certain things (file
> > > layout,
> > > >> >> etc)
> > > >> >> > that provide a stable interface for people to build derived
> > images
> > > >> from?
> > > >> >> >
> > > >> >> > The path of least resistance to answering these questions is to
> > say
> > > >> that
> > > >> >> > the Docker image is provided in the hopes that it is useful,
> but
> > > that
> > > >> >> it is
> > > >> >> > done via an automated build, without any pre-release testing,
> and
> > > >> >> without
> > > >> >> > any particular guarantees about the 'interface' it provides. If
> > > this
> > > >> is
> > > >> >> the
> > > >> >> > case then I would suggest putting it up on Docker Hub with an
> > > >> >> appropriate
> > > >> >> > disclaimer and not promoting it too much. (We might very well
> end
> > > up
> > > >> >> > pushing images every once in a while that don't work right, and
> > it
> > > >> would
> > > >> >> > reflect poorly on the project to have those be prominently
> > > >> linked-to.)
> > > >> >> It
> > > >> >> > becomes easier to strengthen these guarantees if we add an
> > > automated
> > > >> >> test
> > > >> >> > suite that we can run before releases which verifies
> > functionality
> > > >> and
> > > >> >> > interface adherence.
> > > >> >> >
> > > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > >> >> <rm...@vmware.com.invalid>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > This is purely a packaging exercise. I don't see a reason to
> > mark
> > > >> >> this as
> > > >> >> > > experimental.
> > > >> >> > >
> > > >> >> > > Rajiv.
> > > >> >> > > ________________________________
> > > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > >> >> > > To: dev@druid.apache.org
> > > >> >> > > Subject: Re: docker build
> > > >> >> > >
> > > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > > >> >> Infrastructure
> > > >> >> > > team and then we'll have some automatic process that'll
> > > >> automatically
> > > >> >> > > update docker-hub with images relating to each release.
> > > >> >> > >
> > > >> >> > > I guess there's two open questions I think we should reach a
> > > >> >> consensus on
> > > >> >> > > (others feel free to add more!).
> > > >> >> > >
> > > >> >> > > - Are we as a community happy to "support" an additional
> > release
> > > >> >> > artefact?
> > > >> >> > > I'm happy to try to incorporate this into my employer's
> testing
> > > >> >> > > infrastructure to help catch any regressions on future
> releases
> > > but
> > > >> >> > that's
> > > >> >> > > just one data point on each release.
> > > >> >> > >
> > > >> >> > > - Along the same vein, do we follow the same process as we do
> > > with
> > > >> new
> > > >> >> > > features and mark this as experimental for some time?
> > > >> >> > >
> > > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > > wrote:
> > > >> >> > >
> > > >> >> > > > Now that
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > > >> >> > > is merged
> > > >> >> > > > (thank you!)
> > > >> >> > > >
> > > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > > >> >> automatically
> > > >> >> > > on a
> > > >> >> > > > 'tag' of the repo.
> > > >> >> > > >
> > > >> >> > > > Once that is done it is much more convenient for folks to
> use
> > > >> this
> > > >> >> > tool.
> > > >> >> > > >
> > > >> >> > > > --don
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
https://github.com/apache/incubator-druid/pull/7296 is now there which
removes the mysql but shows how to re-enable should someone wish.
On Tue, 19 Mar 2019 at 07:19, Don Bowman <do...@agilicus.com> wrote:
> So if I send a PR removing the mysql connector (which comes from maven
> which is itself apache released), can we be done w/ this?
>
> I don't want to get into the fact that it runs on linux and uses bash in
> scripts.
>
> I think there's some confusion in this thread about what people expect
> from container images. We want this to be the standard image in the
> Kubernetes helm chart as an exxample. No one wants to buiild their own
> container and find their own registry to host it.
>
> But if its solely about mysql, I think removing it is fine, the postgres
> connector works at least as well.
>
> On the 1/2/3 list:
>
> #1: I'll do the PR to remove mysql
> #2. I cannot do this, I have no idea what Jira or apache infra is etc.
> #3. I don't think this is worth doing, no one will use it except for
> someone who can trivially do that themself and might want other things in
> there anyway.
>
> so, if someone can help w/ #2 (getting the travis build to push this or
> whatever) i'll do a PR for #1.
> OK?
>
> On Tue, 5 Mar 2019 at 12:59, Charles Allen <ch...@snap.com.invalid>
> wrote:
>
>> Honestly we're at a very strange impasse. On one hand I don't think the
>> ASF
>> project can adopt an official docker image unless ASF legal says its ok.
>> "Official" releases are source code anyways (as my understanding goes),
>> and
>> binary artifacts are convenience things. Unfortunately I do not see a path
>> forward unless some entity is willing to take on a stance similar as
>> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
>> pretty new territory from a legal perspective (the fact that docker images
>> are layers makes it even more interesting).
>>
>> At this point I think the safest thing to do is something that is "no more
>> GPL dependent than other containers in the apache repo", which would mean
>> not adding in GPL binaries. Which means switching to postgres. I don't
>> foresee an aggressive legal stance on this issue, meaning it might take a
>> while as people watch where the industry is going.
>>
>>
>>
>> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>>
>> > where do we stand on this?
>> > the PR is in and accepted, but i feel we need to have this built as
>> part of
>> > the release artifacts and on dockerhub to foster adoption.
>> > if the only issue is the mysql connector i can remove it in favour of
>> the
>> > postgres connector.
>> >
>> >
>> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
>> >
>> > > i can just remove the mysql, the postgres works, i was just assuming
>> > folks
>> > > wanted it.
>> > >
>> > >
>> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
>> > >
>> > >> A discussion is progressing on
>> > >>
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
>> .
>> > It doesn't seem to have
>> > >> got anywhere firm yet.
>> > >>
>> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> > >>
>> > >> > I don't think anything is strictly needed from you at this point,
>> but
>> > >> > things happen when people drive them, and participation in that
>> effort
>> > >> > would help make sure it gets done. I think at this point the tasks
>> on
>> > >> our
>> > >> > end are watching LEGAL-437 for advice (or making it moot by
>> removing
>> > the
>> > >> > MySQL jar), asking Infra to set up automated builds once that is
>> > sorted
>> > >> > out, and building some kind of consensus around how we'll label and
>> > >> promote
>> > >> > the Docker images.
>> > >> >
>> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
>> wrote:
>> > >> >
>> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
>> > >> metadata.
>> > >> >> if this is the case we should consider relfecting postgres as the
>> > >> default
>> > >> >> metadata in the docs.
>> > >> >> however, i think this is mere aggregation under the gpl license,
>> and
>> > >> the
>> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
>> start
>> > >> >> scripts are all bash-specific as an example.
>> > >> >>
>> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
>> get
>> > an
>> > >> >> automated build going into dockerhub, and tagged w/ each release.
>> i
>> > >> think
>> > >> >> this will help adoption.
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
>> wrote:
>> > >> >>
>> > >> >> > First off thanks a lot for your work here Don!!
>> > >> >> >
>> > >> >> > I really do think, though, that we need to be careful about the
>> > >> >> inclusion
>> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
>> > that
>> > >> >> ASF
>> > >> >> > projects should not distribute it as part of binary convenience
>> > >> >> releases:
>> > >> >> >
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
>> .
>> > I think having the
>> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
>> > >> >> > distributing the jar itself, just, essentially, a pointer to
>> how to
>> > >> >> > download it. But if we start offering a prebuilt Docker image,
>> it
>> > is
>> > >> >> less
>> > >> >> > clear to me if that is fine or not. In the interests of
>> resolving
>> > >> this
>> > >> >> > question one way or the other, I opened a question asking about
>> > this
>> > >> >> > specific situation:
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
>> > .
>> > >> >> >
>> > >> >> > About Dylan's questions: my feeling is that we should go ahead
>> and
>> > >> >> enable
>> > >> >> > automated pushes to Docker Hub, and provide some appropriate
>> > language
>> > >> >> > around what people should expect out of it. I don't think
>> > >> >> 'experimental' is
>> > >> >> > the right word, but we should be clear around exactly what
>> contract
>> > >> we
>> > >> >> are
>> > >> >> > adhering to. Is it something people can expect to be published
>> with
>> > >> each
>> > >> >> > release? Is it something that we are going to build and test as
>> > part
>> > >> of
>> > >> >> the
>> > >> >> > release process, or are we going to publish it via automation
>> > without
>> > >> >> any
>> > >> >> > testing? Is it something we expect people to use in production,
>> or
>> > >> >> > something we only expect people to use for evaluation? If it is
>> > >> >> something
>> > >> >> > we expect people to use in production, do we expect them to use
>> it
>> > in
>> > >> >> any
>> > >> >> > particular way? Will we be guaranteeing certain things (file
>> > layout,
>> > >> >> etc)
>> > >> >> > that provide a stable interface for people to build derived
>> images
>> > >> from?
>> > >> >> >
>> > >> >> > The path of least resistance to answering these questions is to
>> say
>> > >> that
>> > >> >> > the Docker image is provided in the hopes that it is useful, but
>> > that
>> > >> >> it is
>> > >> >> > done via an automated build, without any pre-release testing,
>> and
>> > >> >> without
>> > >> >> > any particular guarantees about the 'interface' it provides. If
>> > this
>> > >> is
>> > >> >> the
>> > >> >> > case then I would suggest putting it up on Docker Hub with an
>> > >> >> appropriate
>> > >> >> > disclaimer and not promoting it too much. (We might very well
>> end
>> > up
>> > >> >> > pushing images every once in a while that don't work right, and
>> it
>> > >> would
>> > >> >> > reflect poorly on the project to have those be prominently
>> > >> linked-to.)
>> > >> >> It
>> > >> >> > becomes easier to strengthen these guarantees if we add an
>> > automated
>> > >> >> test
>> > >> >> > suite that we can run before releases which verifies
>> functionality
>> > >> and
>> > >> >> > interface adherence.
>> > >> >> >
>> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
>> > >> >> <rm...@vmware.com.invalid>
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > This is purely a packaging exercise. I don't see a reason to
>> mark
>> > >> >> this as
>> > >> >> > > experimental.
>> > >> >> > >
>> > >> >> > > Rajiv.
>> > >> >> > > ________________________________
>> > >> >> > > From: Dylan Wylie <dy...@apache.org>
>> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
>> > >> >> > > To: dev@druid.apache.org
>> > >> >> > > Subject: Re: docker build
>> > >> >> > >
>> > >> >> > > I believe all we have to do is submit a ticket to Apache's
>> > >> >> Infrastructure
>> > >> >> > > team and then we'll have some automatic process that'll
>> > >> automatically
>> > >> >> > > update docker-hub with images relating to each release.
>> > >> >> > >
>> > >> >> > > I guess there's two open questions I think we should reach a
>> > >> >> consensus on
>> > >> >> > > (others feel free to add more!).
>> > >> >> > >
>> > >> >> > > - Are we as a community happy to "support" an additional
>> release
>> > >> >> > artefact?
>> > >> >> > > I'm happy to try to incorporate this into my employer's
>> testing
>> > >> >> > > infrastructure to help catch any regressions on future
>> releases
>> > but
>> > >> >> > that's
>> > >> >> > > just one data point on each release.
>> > >> >> > >
>> > >> >> > > - Along the same vein, do we follow the same process as we do
>> > with
>> > >> new
>> > >> >> > > features and mark this as experimental for some time?
>> > >> >> > >
>> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
>> > wrote:
>> > >> >> > >
>> > >> >> > > > Now that
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >>
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
>> > >> >> > > is merged
>> > >> >> > > > (thank you!)
>> > >> >> > > >
>> > >> >> > > > who can get this set to build into Dockerhub? Presumably
>> > >> >> automatically
>> > >> >> > > on a
>> > >> >> > > > 'tag' of the repo.
>> > >> >> > > >
>> > >> >> > > > Once that is done it is much more convenient for folks to
>> use
>> > >> this
>> > >> >> > tool.
>> > >> >> > > >
>> > >> >> > > > --don
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
So if I send a PR removing the mysql connector (which comes from maven
which is itself apache released), can we be done w/ this?
I don't want to get into the fact that it runs on linux and uses bash in
scripts.
I think there's some confusion in this thread about what people expect from
container images. We want this to be the standard image in the Kubernetes
helm chart as an exxample. No one wants to buiild their own container and
find their own registry to host it.
But if its solely about mysql, I think removing it is fine, the postgres
connector works at least as well.
On the 1/2/3 list:
#1: I'll do the PR to remove mysql
#2. I cannot do this, I have no idea what Jira or apache infra is etc.
#3. I don't think this is worth doing, no one will use it except for
someone who can trivially do that themself and might want other things in
there anyway.
so, if someone can help w/ #2 (getting the travis build to push this or
whatever) i'll do a PR for #1.
OK?
On Tue, 5 Mar 2019 at 12:59, Charles Allen <ch...@snap.com.invalid>
wrote:
> Honestly we're at a very strange impasse. On one hand I don't think the ASF
> project can adopt an official docker image unless ASF legal says its ok.
> "Official" releases are source code anyways (as my understanding goes), and
> binary artifacts are convenience things. Unfortunately I do not see a path
> forward unless some entity is willing to take on a stance similar as
> outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
> pretty new territory from a legal perspective (the fact that docker images
> are layers makes it even more interesting).
>
> At this point I think the safest thing to do is something that is "no more
> GPL dependent than other containers in the apache repo", which would mean
> not adding in GPL binaries. Which means switching to postgres. I don't
> foresee an aggressive legal stance on this issue, meaning it might take a
> while as people watch where the industry is going.
>
>
>
> On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
>
> > where do we stand on this?
> > the PR is in and accepted, but i feel we need to have this built as part
> of
> > the release artifacts and on dockerhub to foster adoption.
> > if the only issue is the mysql connector i can remove it in favour of the
> > postgres connector.
> >
> >
> > On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> >
> > > i can just remove the mysql, the postgres works, i was just assuming
> > folks
> > > wanted it.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> A discussion is progressing on
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> > It doesn't seem to have
> > >> got anywhere firm yet.
> > >>
> > >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > I don't think anything is strictly needed from you at this point,
> but
> > >> > things happen when people drive them, and participation in that
> effort
> > >> > would help make sure it gets done. I think at this point the tasks
> on
> > >> our
> > >> > end are watching LEGAL-437 for advice (or making it moot by removing
> > the
> > >> > MySQL jar), asking Infra to set up automated builds once that is
> > sorted
> > >> > out, and building some kind of consensus around how we'll label and
> > >> promote
> > >> > the Docker images.
> > >> >
> > >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> wrote:
> > >> >
> > >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > >> metadata.
> > >> >> if this is the case we should consider relfecting postgres as the
> > >> default
> > >> >> metadata in the docs.
> > >> >> however, i think this is mere aggregation under the gpl license,
> and
> > >> the
> > >> >> docker image tends to have other (e.g. bash) gpl code. druid's
> start
> > >> >> scripts are all bash-specific as an example.
> > >> >>
> > >> >> I'm not clear if anything further is needed of me, i'm hoping to
> get
> > an
> > >> >> automated build going into dockerhub, and tagged w/ each release. i
> > >> think
> > >> >> this will help adoption.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > >> >>
> > >> >> > First off thanks a lot for your work here Don!!
> > >> >> >
> > >> >> > I really do think, though, that we need to be careful about the
> > >> >> inclusion
> > >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> > that
> > >> >> ASF
> > >> >> > projects should not distribute it as part of binary convenience
> > >> >> releases:
> > >> >> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=
> .
> > I think having the
> > >> >> > Dockerfile in the repo is probably fine: in that case we are not
> > >> >> > distributing the jar itself, just, essentially, a pointer to how
> to
> > >> >> > download it. But if we start offering a prebuilt Docker image, it
> > is
> > >> >> less
> > >> >> > clear to me if that is fine or not. In the interests of resolving
> > >> this
> > >> >> > question one way or the other, I opened a question asking about
> > this
> > >> >> > specific situation:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> > .
> > >> >> >
> > >> >> > About Dylan's questions: my feeling is that we should go ahead
> and
> > >> >> enable
> > >> >> > automated pushes to Docker Hub, and provide some appropriate
> > language
> > >> >> > around what people should expect out of it. I don't think
> > >> >> 'experimental' is
> > >> >> > the right word, but we should be clear around exactly what
> contract
> > >> we
> > >> >> are
> > >> >> > adhering to. Is it something people can expect to be published
> with
> > >> each
> > >> >> > release? Is it something that we are going to build and test as
> > part
> > >> of
> > >> >> the
> > >> >> > release process, or are we going to publish it via automation
> > without
> > >> >> any
> > >> >> > testing? Is it something we expect people to use in production,
> or
> > >> >> > something we only expect people to use for evaluation? If it is
> > >> >> something
> > >> >> > we expect people to use in production, do we expect them to use
> it
> > in
> > >> >> any
> > >> >> > particular way? Will we be guaranteeing certain things (file
> > layout,
> > >> >> etc)
> > >> >> > that provide a stable interface for people to build derived
> images
> > >> from?
> > >> >> >
> > >> >> > The path of least resistance to answering these questions is to
> say
> > >> that
> > >> >> > the Docker image is provided in the hopes that it is useful, but
> > that
> > >> >> it is
> > >> >> > done via an automated build, without any pre-release testing, and
> > >> >> without
> > >> >> > any particular guarantees about the 'interface' it provides. If
> > this
> > >> is
> > >> >> the
> > >> >> > case then I would suggest putting it up on Docker Hub with an
> > >> >> appropriate
> > >> >> > disclaimer and not promoting it too much. (We might very well end
> > up
> > >> >> > pushing images every once in a while that don't work right, and
> it
> > >> would
> > >> >> > reflect poorly on the project to have those be prominently
> > >> linked-to.)
> > >> >> It
> > >> >> > becomes easier to strengthen these guarantees if we add an
> > automated
> > >> >> test
> > >> >> > suite that we can run before releases which verifies
> functionality
> > >> and
> > >> >> > interface adherence.
> > >> >> >
> > >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > >> >> <rm...@vmware.com.invalid>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > This is purely a packaging exercise. I don't see a reason to
> mark
> > >> >> this as
> > >> >> > > experimental.
> > >> >> > >
> > >> >> > > Rajiv.
> > >> >> > > ________________________________
> > >> >> > > From: Dylan Wylie <dy...@apache.org>
> > >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > >> >> > > To: dev@druid.apache.org
> > >> >> > > Subject: Re: docker build
> > >> >> > >
> > >> >> > > I believe all we have to do is submit a ticket to Apache's
> > >> >> Infrastructure
> > >> >> > > team and then we'll have some automatic process that'll
> > >> automatically
> > >> >> > > update docker-hub with images relating to each release.
> > >> >> > >
> > >> >> > > I guess there's two open questions I think we should reach a
> > >> >> consensus on
> > >> >> > > (others feel free to add more!).
> > >> >> > >
> > >> >> > > - Are we as a community happy to "support" an additional
> release
> > >> >> > artefact?
> > >> >> > > I'm happy to try to incorporate this into my employer's testing
> > >> >> > > infrastructure to help catch any regressions on future releases
> > but
> > >> >> > that's
> > >> >> > > just one data point on each release.
> > >> >> > >
> > >> >> > > - Along the same vein, do we follow the same process as we do
> > with
> > >> new
> > >> >> > > features and mark this as experimental for some time?
> > >> >> > >
> > >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > wrote:
> > >> >> > >
> > >> >> > > > Now that
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> > >> >> > > is merged
> > >> >> > > > (thank you!)
> > >> >> > > >
> > >> >> > > > who can get this set to build into Dockerhub? Presumably
> > >> >> automatically
> > >> >> > > on a
> > >> >> > > > 'tag' of the repo.
> > >> >> > > >
> > >> >> > > > Once that is done it is much more convenient for folks to use
> > >> this
> > >> >> > tool.
> > >> >> > > >
> > >> >> > > > --don
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
Honestly we're at a very strange impasse. On one hand I don't think the ASF
project can adopt an official docker image unless ASF legal says its ok.
"Official" releases are source code anyways (as my understanding goes), and
binary artifacts are convenience things. Unfortunately I do not see a path
forward unless some entity is willing to take on a stance similar as
outlined in https://issues.apache.org/jira/browse/LEGAL-437 . This is
pretty new territory from a legal perspective (the fact that docker images
are layers makes it even more interesting).
At this point I think the safest thing to do is something that is "no more
GPL dependent than other containers in the apache repo", which would mean
not adding in GPL binaries. Which means switching to postgres. I don't
foresee an aggressive legal stance on this issue, meaning it might take a
while as people watch where the industry is going.
On Tue, Mar 5, 2019 at 8:20 AM Don Bowman <do...@agilicus.com> wrote:
> where do we stand on this?
> the PR is in and accepted, but i feel we need to have this built as part of
> the release artifacts and on dockerhub to foster adoption.
> if the only issue is the mysql connector i can remove it in favour of the
> postgres connector.
>
>
> On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
>
> > i can just remove the mysql, the postgres works, i was just assuming
> folks
> > wanted it.
> >
> >
> > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> >
> >> A discussion is progressing on
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=.
> It doesn't seem to have
> >> got anywhere firm yet.
> >>
> >> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > I don't think anything is strictly needed from you at this point, but
> >> > things happen when people drive them, and participation in that effort
> >> > would help make sure it gets done. I think at this point the tasks on
> >> our
> >> > end are watching LEGAL-437 for advice (or making it moot by removing
> the
> >> > MySQL jar), asking Infra to set up automated builds once that is
> sorted
> >> > out, and building some kind of consensus around how we'll label and
> >> promote
> >> > the Docker images.
> >> >
> >> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> >> >
> >> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> >> metadata.
> >> >> if this is the case we should consider relfecting postgres as the
> >> default
> >> >> metadata in the docs.
> >> >> however, i think this is mere aggregation under the gpl license, and
> >> the
> >> >> docker image tends to have other (e.g. bash) gpl code. druid's start
> >> >> scripts are all bash-specific as an example.
> >> >>
> >> >> I'm not clear if anything further is needed of me, i'm hoping to get
> an
> >> >> automated build going into dockerhub, and tagged w/ each release. i
> >> think
> >> >> this will help adoption.
> >> >>
> >> >>
> >> >>
> >> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> >> >>
> >> >> > First off thanks a lot for your work here Don!!
> >> >> >
> >> >> > I really do think, though, that we need to be careful about the
> >> >> inclusion
> >> >> > of the MySQL connector jar. ASF legal has been clear in the past
> that
> >> >> ASF
> >> >> > projects should not distribute it as part of binary convenience
> >> >> releases:
> >> >> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D200&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=tzsmBm2IIaa5BS9lQrTc9e0GDt09RmMiI4gfn9CoHT4&e=.
> I think having the
> >> >> > Dockerfile in the repo is probably fine: in that case we are not
> >> >> > distributing the jar itself, just, essentially, a pointer to how to
> >> >> > download it. But if we start offering a prebuilt Docker image, it
> is
> >> >> less
> >> >> > clear to me if that is fine or not. In the interests of resolving
> >> this
> >> >> > question one way or the other, I opened a question asking about
> this
> >> >> > specific situation:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LEGAL-2D437&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=uXRQeP8EnjcHXQmG5oJoTQN2ztfu7N0YCNLpS_aj93g&e=
> .
> >> >> >
> >> >> > About Dylan's questions: my feeling is that we should go ahead and
> >> >> enable
> >> >> > automated pushes to Docker Hub, and provide some appropriate
> language
> >> >> > around what people should expect out of it. I don't think
> >> >> 'experimental' is
> >> >> > the right word, but we should be clear around exactly what contract
> >> we
> >> >> are
> >> >> > adhering to. Is it something people can expect to be published with
> >> each
> >> >> > release? Is it something that we are going to build and test as
> part
> >> of
> >> >> the
> >> >> > release process, or are we going to publish it via automation
> without
> >> >> any
> >> >> > testing? Is it something we expect people to use in production, or
> >> >> > something we only expect people to use for evaluation? If it is
> >> >> something
> >> >> > we expect people to use in production, do we expect them to use it
> in
> >> >> any
> >> >> > particular way? Will we be guaranteeing certain things (file
> layout,
> >> >> etc)
> >> >> > that provide a stable interface for people to build derived images
> >> from?
> >> >> >
> >> >> > The path of least resistance to answering these questions is to say
> >> that
> >> >> > the Docker image is provided in the hopes that it is useful, but
> that
> >> >> it is
> >> >> > done via an automated build, without any pre-release testing, and
> >> >> without
> >> >> > any particular guarantees about the 'interface' it provides. If
> this
> >> is
> >> >> the
> >> >> > case then I would suggest putting it up on Docker Hub with an
> >> >> appropriate
> >> >> > disclaimer and not promoting it too much. (We might very well end
> up
> >> >> > pushing images every once in a while that don't work right, and it
> >> would
> >> >> > reflect poorly on the project to have those be prominently
> >> linked-to.)
> >> >> It
> >> >> > becomes easier to strengthen these guarantees if we add an
> automated
> >> >> test
> >> >> > suite that we can run before releases which verifies functionality
> >> and
> >> >> > interface adherence.
> >> >> >
> >> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> >> >> <rm...@vmware.com.invalid>
> >> >> > wrote:
> >> >> >
> >> >> > > This is purely a packaging exercise. I don't see a reason to mark
> >> >> this as
> >> >> > > experimental.
> >> >> > >
> >> >> > > Rajiv.
> >> >> > > ________________________________
> >> >> > > From: Dylan Wylie <dy...@apache.org>
> >> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> >> >> > > To: dev@druid.apache.org
> >> >> > > Subject: Re: docker build
> >> >> > >
> >> >> > > I believe all we have to do is submit a ticket to Apache's
> >> >> Infrastructure
> >> >> > > team and then we'll have some automatic process that'll
> >> automatically
> >> >> > > update docker-hub with images relating to each release.
> >> >> > >
> >> >> > > I guess there's two open questions I think we should reach a
> >> >> consensus on
> >> >> > > (others feel free to add more!).
> >> >> > >
> >> >> > > - Are we as a community happy to "support" an additional release
> >> >> > artefact?
> >> >> > > I'm happy to try to incorporate this into my employer's testing
> >> >> > > infrastructure to help catch any regressions on future releases
> but
> >> >> > that's
> >> >> > > just one data point on each release.
> >> >> > >
> >> >> > > - Along the same vein, do we follow the same process as we do
> with
> >> new
> >> >> > > features and mark this as experimental for some time?
> >> >> > >
> >> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> wrote:
> >> >> > >
> >> >> > > > Now that
> >> >> > >
> >> >> >
> >> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fincubator-2Ddruid-252Fpull-252F6896-26amp-3Bdata-3D02-257C01-257Crmordani-2540vmware.com-257C942b2af1dfb740fcbed308d68dcef937-257Cb39138ca3cee4b4aa4d6cd83d9dd62f0-257C0-257C1-257C636852317419449405-26amp-3Bsdata-3DEXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=SDcL2cv8y5vfiK64aTakmAF8xiMVateJ6QQ3JTsegRI&s=I0Jmt-SqSpozqPWg-3MxWry3mQC_oFP2v9FhGUTL5Ls&e=
> >> >> > > is merged
> >> >> > > > (thank you!)
> >> >> > > >
> >> >> > > > who can get this set to build into Dockerhub? Presumably
> >> >> automatically
> >> >> > > on a
> >> >> > > > 'tag' of the repo.
> >> >> > > >
> >> >> > > > Once that is done it is much more convenient for folks to use
> >> this
> >> >> > tool.
> >> >> > > >
> >> >> > > > --don
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
where do we stand on this?
the PR is in and accepted, but i feel we need to have this built as part of
the release artifacts and on dockerhub to foster adoption.
if the only issue is the mysql connector i can remove it in favour of the
postgres connector.
On Mon, 18 Feb 2019 at 13:58, Don Bowman <do...@agilicus.com> wrote:
> i can just remove the mysql, the postgres works, i was just assuming folks
> wanted it.
>
>
> On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
>
>> A discussion is progressing on
>> https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to have
>> got anywhere firm yet.
>>
>> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > I don't think anything is strictly needed from you at this point, but
>> > things happen when people drive them, and participation in that effort
>> > would help make sure it gets done. I think at this point the tasks on
>> our
>> > end are watching LEGAL-437 for advice (or making it moot by removing the
>> > MySQL jar), asking Infra to set up automated builds once that is sorted
>> > out, and building some kind of consensus around how we'll label and
>> promote
>> > the Docker images.
>> >
>> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
>> >
>> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
>> metadata.
>> >> if this is the case we should consider relfecting postgres as the
>> default
>> >> metadata in the docs.
>> >> however, i think this is mere aggregation under the gpl license, and
>> the
>> >> docker image tends to have other (e.g. bash) gpl code. druid's start
>> >> scripts are all bash-specific as an example.
>> >>
>> >> I'm not clear if anything further is needed of me, i'm hoping to get an
>> >> automated build going into dockerhub, and tagged w/ each release. i
>> think
>> >> this will help adoption.
>> >>
>> >>
>> >>
>> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
>> >>
>> >> > First off thanks a lot for your work here Don!!
>> >> >
>> >> > I really do think, though, that we need to be careful about the
>> >> inclusion
>> >> > of the MySQL connector jar. ASF legal has been clear in the past that
>> >> ASF
>> >> > projects should not distribute it as part of binary convenience
>> >> releases:
>> >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
>> >> > Dockerfile in the repo is probably fine: in that case we are not
>> >> > distributing the jar itself, just, essentially, a pointer to how to
>> >> > download it. But if we start offering a prebuilt Docker image, it is
>> >> less
>> >> > clear to me if that is fine or not. In the interests of resolving
>> this
>> >> > question one way or the other, I opened a question asking about this
>> >> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
>> >> >
>> >> > About Dylan's questions: my feeling is that we should go ahead and
>> >> enable
>> >> > automated pushes to Docker Hub, and provide some appropriate language
>> >> > around what people should expect out of it. I don't think
>> >> 'experimental' is
>> >> > the right word, but we should be clear around exactly what contract
>> we
>> >> are
>> >> > adhering to. Is it something people can expect to be published with
>> each
>> >> > release? Is it something that we are going to build and test as part
>> of
>> >> the
>> >> > release process, or are we going to publish it via automation without
>> >> any
>> >> > testing? Is it something we expect people to use in production, or
>> >> > something we only expect people to use for evaluation? If it is
>> >> something
>> >> > we expect people to use in production, do we expect them to use it in
>> >> any
>> >> > particular way? Will we be guaranteeing certain things (file layout,
>> >> etc)
>> >> > that provide a stable interface for people to build derived images
>> from?
>> >> >
>> >> > The path of least resistance to answering these questions is to say
>> that
>> >> > the Docker image is provided in the hopes that it is useful, but that
>> >> it is
>> >> > done via an automated build, without any pre-release testing, and
>> >> without
>> >> > any particular guarantees about the 'interface' it provides. If this
>> is
>> >> the
>> >> > case then I would suggest putting it up on Docker Hub with an
>> >> appropriate
>> >> > disclaimer and not promoting it too much. (We might very well end up
>> >> > pushing images every once in a while that don't work right, and it
>> would
>> >> > reflect poorly on the project to have those be prominently
>> linked-to.)
>> >> It
>> >> > becomes easier to strengthen these guarantees if we add an automated
>> >> test
>> >> > suite that we can run before releases which verifies functionality
>> and
>> >> > interface adherence.
>> >> >
>> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
>> >> <rm...@vmware.com.invalid>
>> >> > wrote:
>> >> >
>> >> > > This is purely a packaging exercise. I don't see a reason to mark
>> >> this as
>> >> > > experimental.
>> >> > >
>> >> > > Rajiv.
>> >> > > ________________________________
>> >> > > From: Dylan Wylie <dy...@apache.org>
>> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
>> >> > > To: dev@druid.apache.org
>> >> > > Subject: Re: docker build
>> >> > >
>> >> > > I believe all we have to do is submit a ticket to Apache's
>> >> Infrastructure
>> >> > > team and then we'll have some automatic process that'll
>> automatically
>> >> > > update docker-hub with images relating to each release.
>> >> > >
>> >> > > I guess there's two open questions I think we should reach a
>> >> consensus on
>> >> > > (others feel free to add more!).
>> >> > >
>> >> > > - Are we as a community happy to "support" an additional release
>> >> > artefact?
>> >> > > I'm happy to try to incorporate this into my employer's testing
>> >> > > infrastructure to help catch any regressions on future releases but
>> >> > that's
>> >> > > just one data point on each release.
>> >> > >
>> >> > > - Along the same vein, do we follow the same process as we do with
>> new
>> >> > > features and mark this as experimental for some time?
>> >> > >
>> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
>> >> > >
>> >> > > > Now that
>> >> > >
>> >> >
>> >>
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
>> >> > > is merged
>> >> > > > (thank you!)
>> >> > > >
>> >> > > > who can get this set to build into Dockerhub? Presumably
>> >> automatically
>> >> > > on a
>> >> > > > 'tag' of the repo.
>> >> > > >
>> >> > > > Once that is done it is much more convenient for folks to use
>> this
>> >> > tool.
>> >> > > >
>> >> > > > --don
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
Since I accidentally injected some confusion I'll take it a step further.
We have doubly hermetically sealed environments. one docker file for the
Build and then it copies over to a minimalist image for the final
deployable image (using FROM and COPY --from ) .
Some of our tests are docker based as well to make sure there are not
conflicts in different places it might be run.
On Tue, Feb 19, 2019 at 7:47 AM Don Bowman <do...@agilicus.com> wrote:
> On Mon, 18 Feb 2019 at 19:14, Gaurav Bhatnagar <ga...@gmail.com> wrote:
>
> > I have been thinking if automated scripts can be provided to end users in
> > Druid for "Additional Dependencies" for user initiated installation and
> > configuration of optional dependencies to avoid licensing issues. Later
> > these scripts can be integrated in admin UI as configuration wizards.
> >
> >
> >
> Personally I think this is the opposite way the universe is going.
> People want 'hermetic' images w/ read-only filesystems, named by a single
> tag or SHA hash. This is what the container universe is about.
> There's some work to do in druid (e.g. middlemanager logs) to improve this
> (it currently logs into files in there rather than stdout by default, and
> expects that elsewhere).
>
> w/ a product of the scale of druid, its unlikely to be targetted @ 'small'
> deployments.
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
haha, sorry, I mean I agree with Don
On Tue, Mar 5, 2019 at 9:37 AM Charles Allen <ch...@snap.com> wrote:
> I agree with Gian on this sentiment.
>
> On Tue, Feb 19, 2019 at 7:47 AM Don Bowman <do...@agilicus.com> wrote:
>
>> On Mon, 18 Feb 2019 at 19:14, Gaurav Bhatnagar <ga...@gmail.com>
>> wrote:
>>
>> > I have been thinking if automated scripts can be provided to end users
>> in
>> > Druid for "Additional Dependencies" for user initiated installation and
>> > configuration of optional dependencies to avoid licensing issues. Later
>> > these scripts can be integrated in admin UI as configuration wizards.
>> >
>> >
>> >
>> Personally I think this is the opposite way the universe is going.
>> People want 'hermetic' images w/ read-only filesystems, named by a single
>> tag or SHA hash. This is what the container universe is about.
>> There's some work to do in druid (e.g. middlemanager logs) to improve this
>> (it currently logs into files in there rather than stdout by default, and
>> expects that elsewhere).
>>
>> w/ a product of the scale of druid, its unlikely to be targetted @ 'small'
>> deployments.
>>
>
Re: docker build
Posted by Charles Allen <ch...@snap.com.INVALID>.
I agree with Gian on this sentiment.
On Tue, Feb 19, 2019 at 7:47 AM Don Bowman <do...@agilicus.com> wrote:
> On Mon, 18 Feb 2019 at 19:14, Gaurav Bhatnagar <ga...@gmail.com> wrote:
>
> > I have been thinking if automated scripts can be provided to end users in
> > Druid for "Additional Dependencies" for user initiated installation and
> > configuration of optional dependencies to avoid licensing issues. Later
> > these scripts can be integrated in admin UI as configuration wizards.
> >
> >
> >
> Personally I think this is the opposite way the universe is going.
> People want 'hermetic' images w/ read-only filesystems, named by a single
> tag or SHA hash. This is what the container universe is about.
> There's some work to do in druid (e.g. middlemanager logs) to improve this
> (it currently logs into files in there rather than stdout by default, and
> expects that elsewhere).
>
> w/ a product of the scale of druid, its unlikely to be targetted @ 'small'
> deployments.
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
On Mon, 18 Feb 2019 at 19:14, Gaurav Bhatnagar <ga...@gmail.com> wrote:
> I have been thinking if automated scripts can be provided to end users in
> Druid for "Additional Dependencies" for user initiated installation and
> configuration of optional dependencies to avoid licensing issues. Later
> these scripts can be integrated in admin UI as configuration wizards.
>
>
>
Personally I think this is the opposite way the universe is going.
People want 'hermetic' images w/ read-only filesystems, named by a single
tag or SHA hash. This is what the container universe is about.
There's some work to do in druid (e.g. middlemanager logs) to improve this
(it currently logs into files in there rather than stdout by default, and
expects that elsewhere).
w/ a product of the scale of druid, its unlikely to be targetted @ 'small'
deployments.
Re: docker build
Posted by Gaurav Bhatnagar <ga...@gmail.com>.
I have been thinking if automated scripts can be provided to end users in
Druid for "Additional Dependencies" for user initiated installation and
configuration of optional dependencies to avoid licensing issues. Later
these scripts can be integrated in admin UI as configuration wizards.
On Mon, Feb 18, 2019 at 2:07 PM Don Bowman <do...@agilicus.com> wrote:
> reading that link its not clear that mysql is the only concern given that
> bash et al is present.
> but druid is written such that it requires bash (all the run-scripts are
> bash-specific, not sh).
>
> i guess i'll wait a bit to see what happens.
>
> On Mon, 18 Feb 2019 at 17:03, Gian Merlino <gi...@apache.org> wrote:
>
> > I think people definitely do want it, but since the connector is GPL and
> > GPL is Category X, we need to dot our i's and cross our t's when
> > considering whether / how to include it. Removing it pending the
> completion
> > of that discussion is definitely the path of least resistance.
> >
> > On Mon, Feb 18, 2019 at 1:59 PM Don Bowman <do...@agilicus.com> wrote:
> >
> > > i can just remove the mysql, the postgres works, i was just assuming
> > folks
> > > wanted it.
> > >
> > >
> > > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > A discussion is progressing on
> > > > https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to
> > have
> > > > got anywhere firm yet.
> > > >
> > > > On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > I don't think anything is strictly needed from you at this point,
> but
> > > > > things happen when people drive them, and participation in that
> > effort
> > > > > would help make sure it gets done. I think at this point the tasks
> on
> > > our
> > > > > end are watching LEGAL-437 for advice (or making it moot by
> removing
> > > the
> > > > > MySQL jar), asking Infra to set up automated builds once that is
> > sorted
> > > > > out, and building some kind of consensus around how we'll label and
> > > > promote
> > > > > the Docker images.
> > > > >
> > > > > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com>
> wrote:
> > > > >
> > > > >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > > metadata.
> > > > >> if this is the case we should consider relfecting postgres as the
> > > > default
> > > > >> metadata in the docs.
> > > > >> however, i think this is mere aggregation under the gpl license,
> and
> > > the
> > > > >> docker image tends to have other (e.g. bash) gpl code. druid's
> start
> > > > >> scripts are all bash-specific as an example.
> > > > >>
> > > > >> I'm not clear if anything further is needed of me, i'm hoping to
> get
> > > an
> > > > >> automated build going into dockerhub, and tagged w/ each release.
> i
> > > > think
> > > > >> this will help adoption.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org>
> wrote:
> > > > >>
> > > > >> > First off thanks a lot for your work here Don!!
> > > > >> >
> > > > >> > I really do think, though, that we need to be careful about the
> > > > >> inclusion
> > > > >> > of the MySQL connector jar. ASF legal has been clear in the past
> > > that
> > > > >> ASF
> > > > >> > projects should not distribute it as part of binary convenience
> > > > >> releases:
> > > > >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having
> > the
> > > > >> > Dockerfile in the repo is probably fine: in that case we are not
> > > > >> > distributing the jar itself, just, essentially, a pointer to how
> > to
> > > > >> > download it. But if we start offering a prebuilt Docker image,
> it
> > is
> > > > >> less
> > > > >> > clear to me if that is fine or not. In the interests of
> resolving
> > > this
> > > > >> > question one way or the other, I opened a question asking about
> > this
> > > > >> > specific situation:
> > https://issues.apache.org/jira/browse/LEGAL-437
> > > .
> > > > >> >
> > > > >> > About Dylan's questions: my feeling is that we should go ahead
> and
> > > > >> enable
> > > > >> > automated pushes to Docker Hub, and provide some appropriate
> > > language
> > > > >> > around what people should expect out of it. I don't think
> > > > >> 'experimental' is
> > > > >> > the right word, but we should be clear around exactly what
> > contract
> > > we
> > > > >> are
> > > > >> > adhering to. Is it something people can expect to be published
> > with
> > > > each
> > > > >> > release? Is it something that we are going to build and test as
> > part
> > > > of
> > > > >> the
> > > > >> > release process, or are we going to publish it via automation
> > > without
> > > > >> any
> > > > >> > testing? Is it something we expect people to use in production,
> or
> > > > >> > something we only expect people to use for evaluation? If it is
> > > > >> something
> > > > >> > we expect people to use in production, do we expect them to use
> it
> > > in
> > > > >> any
> > > > >> > particular way? Will we be guaranteeing certain things (file
> > layout,
> > > > >> etc)
> > > > >> > that provide a stable interface for people to build derived
> images
> > > > from?
> > > > >> >
> > > > >> > The path of least resistance to answering these questions is to
> > say
> > > > that
> > > > >> > the Docker image is provided in the hopes that it is useful, but
> > > that
> > > > >> it is
> > > > >> > done via an automated build, without any pre-release testing,
> and
> > > > >> without
> > > > >> > any particular guarantees about the 'interface' it provides. If
> > this
> > > > is
> > > > >> the
> > > > >> > case then I would suggest putting it up on Docker Hub with an
> > > > >> appropriate
> > > > >> > disclaimer and not promoting it too much. (We might very well
> end
> > up
> > > > >> > pushing images every once in a while that don't work right, and
> it
> > > > would
> > > > >> > reflect poorly on the project to have those be prominently
> > > linked-to.)
> > > > >> It
> > > > >> > becomes easier to strengthen these guarantees if we add an
> > automated
> > > > >> test
> > > > >> > suite that we can run before releases which verifies
> functionality
> > > and
> > > > >> > interface adherence.
> > > > >> >
> > > > >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > > >> <rm...@vmware.com.invalid>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > This is purely a packaging exercise. I don't see a reason to
> > mark
> > > > >> this as
> > > > >> > > experimental.
> > > > >> > >
> > > > >> > > Rajiv.
> > > > >> > > ________________________________
> > > > >> > > From: Dylan Wylie <dy...@apache.org>
> > > > >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > > >> > > To: dev@druid.apache.org
> > > > >> > > Subject: Re: docker build
> > > > >> > >
> > > > >> > > I believe all we have to do is submit a ticket to Apache's
> > > > >> Infrastructure
> > > > >> > > team and then we'll have some automatic process that'll
> > > > automatically
> > > > >> > > update docker-hub with images relating to each release.
> > > > >> > >
> > > > >> > > I guess there's two open questions I think we should reach a
> > > > >> consensus on
> > > > >> > > (others feel free to add more!).
> > > > >> > >
> > > > >> > > - Are we as a community happy to "support" an additional
> release
> > > > >> > artefact?
> > > > >> > > I'm happy to try to incorporate this into my employer's
> testing
> > > > >> > > infrastructure to help catch any regressions on future
> releases
> > > but
> > > > >> > that's
> > > > >> > > just one data point on each release.
> > > > >> > >
> > > > >> > > - Along the same vein, do we follow the same process as we do
> > with
> > > > new
> > > > >> > > features and mark this as experimental for some time?
> > > > >> > >
> > > > >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> > wrote:
> > > > >> > >
> > > > >> > > > Now that
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> > > > >> > > is merged
> > > > >> > > > (thank you!)
> > > > >> > > >
> > > > >> > > > who can get this set to build into Dockerhub? Presumably
> > > > >> automatically
> > > > >> > > on a
> > > > >> > > > 'tag' of the repo.
> > > > >> > > >
> > > > >> > > > Once that is done it is much more convenient for folks to
> use
> > > this
> > > > >> > tool.
> > > > >> > > >
> > > > >> > > > --don
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
reading that link its not clear that mysql is the only concern given that
bash et al is present.
but druid is written such that it requires bash (all the run-scripts are
bash-specific, not sh).
i guess i'll wait a bit to see what happens.
On Mon, 18 Feb 2019 at 17:03, Gian Merlino <gi...@apache.org> wrote:
> I think people definitely do want it, but since the connector is GPL and
> GPL is Category X, we need to dot our i's and cross our t's when
> considering whether / how to include it. Removing it pending the completion
> of that discussion is definitely the path of least resistance.
>
> On Mon, Feb 18, 2019 at 1:59 PM Don Bowman <do...@agilicus.com> wrote:
>
> > i can just remove the mysql, the postgres works, i was just assuming
> folks
> > wanted it.
> >
> >
> > On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> >
> > > A discussion is progressing on
> > > https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to
> have
> > > got anywhere firm yet.
> > >
> > > On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I don't think anything is strictly needed from you at this point, but
> > > > things happen when people drive them, and participation in that
> effort
> > > > would help make sure it gets done. I think at this point the tasks on
> > our
> > > > end are watching LEGAL-437 for advice (or making it moot by removing
> > the
> > > > MySQL jar), asking Infra to set up automated builds once that is
> sorted
> > > > out, and building some kind of consensus around how we'll label and
> > > promote
> > > > the Docker images.
> > > >
> > > > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> > > >
> > > >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > > metadata.
> > > >> if this is the case we should consider relfecting postgres as the
> > > default
> > > >> metadata in the docs.
> > > >> however, i think this is mere aggregation under the gpl license, and
> > the
> > > >> docker image tends to have other (e.g. bash) gpl code. druid's start
> > > >> scripts are all bash-specific as an example.
> > > >>
> > > >> I'm not clear if anything further is needed of me, i'm hoping to get
> > an
> > > >> automated build going into dockerhub, and tagged w/ each release. i
> > > think
> > > >> this will help adoption.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > > >>
> > > >> > First off thanks a lot for your work here Don!!
> > > >> >
> > > >> > I really do think, though, that we need to be careful about the
> > > >> inclusion
> > > >> > of the MySQL connector jar. ASF legal has been clear in the past
> > that
> > > >> ASF
> > > >> > projects should not distribute it as part of binary convenience
> > > >> releases:
> > > >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having
> the
> > > >> > Dockerfile in the repo is probably fine: in that case we are not
> > > >> > distributing the jar itself, just, essentially, a pointer to how
> to
> > > >> > download it. But if we start offering a prebuilt Docker image, it
> is
> > > >> less
> > > >> > clear to me if that is fine or not. In the interests of resolving
> > this
> > > >> > question one way or the other, I opened a question asking about
> this
> > > >> > specific situation:
> https://issues.apache.org/jira/browse/LEGAL-437
> > .
> > > >> >
> > > >> > About Dylan's questions: my feeling is that we should go ahead and
> > > >> enable
> > > >> > automated pushes to Docker Hub, and provide some appropriate
> > language
> > > >> > around what people should expect out of it. I don't think
> > > >> 'experimental' is
> > > >> > the right word, but we should be clear around exactly what
> contract
> > we
> > > >> are
> > > >> > adhering to. Is it something people can expect to be published
> with
> > > each
> > > >> > release? Is it something that we are going to build and test as
> part
> > > of
> > > >> the
> > > >> > release process, or are we going to publish it via automation
> > without
> > > >> any
> > > >> > testing? Is it something we expect people to use in production, or
> > > >> > something we only expect people to use for evaluation? If it is
> > > >> something
> > > >> > we expect people to use in production, do we expect them to use it
> > in
> > > >> any
> > > >> > particular way? Will we be guaranteeing certain things (file
> layout,
> > > >> etc)
> > > >> > that provide a stable interface for people to build derived images
> > > from?
> > > >> >
> > > >> > The path of least resistance to answering these questions is to
> say
> > > that
> > > >> > the Docker image is provided in the hopes that it is useful, but
> > that
> > > >> it is
> > > >> > done via an automated build, without any pre-release testing, and
> > > >> without
> > > >> > any particular guarantees about the 'interface' it provides. If
> this
> > > is
> > > >> the
> > > >> > case then I would suggest putting it up on Docker Hub with an
> > > >> appropriate
> > > >> > disclaimer and not promoting it too much. (We might very well end
> up
> > > >> > pushing images every once in a while that don't work right, and it
> > > would
> > > >> > reflect poorly on the project to have those be prominently
> > linked-to.)
> > > >> It
> > > >> > becomes easier to strengthen these guarantees if we add an
> automated
> > > >> test
> > > >> > suite that we can run before releases which verifies functionality
> > and
> > > >> > interface adherence.
> > > >> >
> > > >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > > >> <rm...@vmware.com.invalid>
> > > >> > wrote:
> > > >> >
> > > >> > > This is purely a packaging exercise. I don't see a reason to
> mark
> > > >> this as
> > > >> > > experimental.
> > > >> > >
> > > >> > > Rajiv.
> > > >> > > ________________________________
> > > >> > > From: Dylan Wylie <dy...@apache.org>
> > > >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > >> > > To: dev@druid.apache.org
> > > >> > > Subject: Re: docker build
> > > >> > >
> > > >> > > I believe all we have to do is submit a ticket to Apache's
> > > >> Infrastructure
> > > >> > > team and then we'll have some automatic process that'll
> > > automatically
> > > >> > > update docker-hub with images relating to each release.
> > > >> > >
> > > >> > > I guess there's two open questions I think we should reach a
> > > >> consensus on
> > > >> > > (others feel free to add more!).
> > > >> > >
> > > >> > > - Are we as a community happy to "support" an additional release
> > > >> > artefact?
> > > >> > > I'm happy to try to incorporate this into my employer's testing
> > > >> > > infrastructure to help catch any regressions on future releases
> > but
> > > >> > that's
> > > >> > > just one data point on each release.
> > > >> > >
> > > >> > > - Along the same vein, do we follow the same process as we do
> with
> > > new
> > > >> > > features and mark this as experimental for some time?
> > > >> > >
> > > >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com>
> wrote:
> > > >> > >
> > > >> > > > Now that
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> > > >> > > is merged
> > > >> > > > (thank you!)
> > > >> > > >
> > > >> > > > who can get this set to build into Dockerhub? Presumably
> > > >> automatically
> > > >> > > on a
> > > >> > > > 'tag' of the repo.
> > > >> > > >
> > > >> > > > Once that is done it is much more convenient for folks to use
> > this
> > > >> > tool.
> > > >> > > >
> > > >> > > > --don
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
I think people definitely do want it, but since the connector is GPL and
GPL is Category X, we need to dot our i's and cross our t's when
considering whether / how to include it. Removing it pending the completion
of that discussion is definitely the path of least resistance.
On Mon, Feb 18, 2019 at 1:59 PM Don Bowman <do...@agilicus.com> wrote:
> i can just remove the mysql, the postgres works, i was just assuming folks
> wanted it.
>
>
> On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
>
> > A discussion is progressing on
> > https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to have
> > got anywhere firm yet.
> >
> > On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I don't think anything is strictly needed from you at this point, but
> > > things happen when people drive them, and participation in that effort
> > > would help make sure it gets done. I think at this point the tasks on
> our
> > > end are watching LEGAL-437 for advice (or making it moot by removing
> the
> > > MySQL jar), asking Infra to set up automated builds once that is sorted
> > > out, and building some kind of consensus around how we'll label and
> > promote
> > > the Docker images.
> > >
> > > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> > >
> > >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> > metadata.
> > >> if this is the case we should consider relfecting postgres as the
> > default
> > >> metadata in the docs.
> > >> however, i think this is mere aggregation under the gpl license, and
> the
> > >> docker image tends to have other (e.g. bash) gpl code. druid's start
> > >> scripts are all bash-specific as an example.
> > >>
> > >> I'm not clear if anything further is needed of me, i'm hoping to get
> an
> > >> automated build going into dockerhub, and tagged w/ each release. i
> > think
> > >> this will help adoption.
> > >>
> > >>
> > >>
> > >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > First off thanks a lot for your work here Don!!
> > >> >
> > >> > I really do think, though, that we need to be careful about the
> > >> inclusion
> > >> > of the MySQL connector jar. ASF legal has been clear in the past
> that
> > >> ASF
> > >> > projects should not distribute it as part of binary convenience
> > >> releases:
> > >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
> > >> > Dockerfile in the repo is probably fine: in that case we are not
> > >> > distributing the jar itself, just, essentially, a pointer to how to
> > >> > download it. But if we start offering a prebuilt Docker image, it is
> > >> less
> > >> > clear to me if that is fine or not. In the interests of resolving
> this
> > >> > question one way or the other, I opened a question asking about this
> > >> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437
> .
> > >> >
> > >> > About Dylan's questions: my feeling is that we should go ahead and
> > >> enable
> > >> > automated pushes to Docker Hub, and provide some appropriate
> language
> > >> > around what people should expect out of it. I don't think
> > >> 'experimental' is
> > >> > the right word, but we should be clear around exactly what contract
> we
> > >> are
> > >> > adhering to. Is it something people can expect to be published with
> > each
> > >> > release? Is it something that we are going to build and test as part
> > of
> > >> the
> > >> > release process, or are we going to publish it via automation
> without
> > >> any
> > >> > testing? Is it something we expect people to use in production, or
> > >> > something we only expect people to use for evaluation? If it is
> > >> something
> > >> > we expect people to use in production, do we expect them to use it
> in
> > >> any
> > >> > particular way? Will we be guaranteeing certain things (file layout,
> > >> etc)
> > >> > that provide a stable interface for people to build derived images
> > from?
> > >> >
> > >> > The path of least resistance to answering these questions is to say
> > that
> > >> > the Docker image is provided in the hopes that it is useful, but
> that
> > >> it is
> > >> > done via an automated build, without any pre-release testing, and
> > >> without
> > >> > any particular guarantees about the 'interface' it provides. If this
> > is
> > >> the
> > >> > case then I would suggest putting it up on Docker Hub with an
> > >> appropriate
> > >> > disclaimer and not promoting it too much. (We might very well end up
> > >> > pushing images every once in a while that don't work right, and it
> > would
> > >> > reflect poorly on the project to have those be prominently
> linked-to.)
> > >> It
> > >> > becomes easier to strengthen these guarantees if we add an automated
> > >> test
> > >> > suite that we can run before releases which verifies functionality
> and
> > >> > interface adherence.
> > >> >
> > >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> > >> <rm...@vmware.com.invalid>
> > >> > wrote:
> > >> >
> > >> > > This is purely a packaging exercise. I don't see a reason to mark
> > >> this as
> > >> > > experimental.
> > >> > >
> > >> > > Rajiv.
> > >> > > ________________________________
> > >> > > From: Dylan Wylie <dy...@apache.org>
> > >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > >> > > To: dev@druid.apache.org
> > >> > > Subject: Re: docker build
> > >> > >
> > >> > > I believe all we have to do is submit a ticket to Apache's
> > >> Infrastructure
> > >> > > team and then we'll have some automatic process that'll
> > automatically
> > >> > > update docker-hub with images relating to each release.
> > >> > >
> > >> > > I guess there's two open questions I think we should reach a
> > >> consensus on
> > >> > > (others feel free to add more!).
> > >> > >
> > >> > > - Are we as a community happy to "support" an additional release
> > >> > artefact?
> > >> > > I'm happy to try to incorporate this into my employer's testing
> > >> > > infrastructure to help catch any regressions on future releases
> but
> > >> > that's
> > >> > > just one data point on each release.
> > >> > >
> > >> > > - Along the same vein, do we follow the same process as we do with
> > new
> > >> > > features and mark this as experimental for some time?
> > >> > >
> > >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> > >> > >
> > >> > > > Now that
> > >> > >
> > >> >
> > >>
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> > >> > > is merged
> > >> > > > (thank you!)
> > >> > > >
> > >> > > > who can get this set to build into Dockerhub? Presumably
> > >> automatically
> > >> > > on a
> > >> > > > 'tag' of the repo.
> > >> > > >
> > >> > > > Once that is done it is much more convenient for folks to use
> this
> > >> > tool.
> > >> > > >
> > >> > > > --don
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
i can just remove the mysql, the postgres works, i was just assuming folks
wanted it.
On Mon, 18 Feb 2019 at 16:58, Gian Merlino <gi...@apache.org> wrote:
> A discussion is progressing on
> https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to have
> got anywhere firm yet.
>
> On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I don't think anything is strictly needed from you at this point, but
> > things happen when people drive them, and participation in that effort
> > would help make sure it gets done. I think at this point the tasks on our
> > end are watching LEGAL-437 for advice (or making it moot by removing the
> > MySQL jar), asking Infra to set up automated builds once that is sorted
> > out, and building some kind of consensus around how we'll label and
> promote
> > the Docker images.
> >
> > On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> >
> >> i'd be fine w/ removing the mysql, i'm using postgresql for the
> metadata.
> >> if this is the case we should consider relfecting postgres as the
> default
> >> metadata in the docs.
> >> however, i think this is mere aggregation under the gpl license, and the
> >> docker image tends to have other (e.g. bash) gpl code. druid's start
> >> scripts are all bash-specific as an example.
> >>
> >> I'm not clear if anything further is needed of me, i'm hoping to get an
> >> automated build going into dockerhub, and tagged w/ each release. i
> think
> >> this will help adoption.
> >>
> >>
> >>
> >> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > First off thanks a lot for your work here Don!!
> >> >
> >> > I really do think, though, that we need to be careful about the
> >> inclusion
> >> > of the MySQL connector jar. ASF legal has been clear in the past that
> >> ASF
> >> > projects should not distribute it as part of binary convenience
> >> releases:
> >> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
> >> > Dockerfile in the repo is probably fine: in that case we are not
> >> > distributing the jar itself, just, essentially, a pointer to how to
> >> > download it. But if we start offering a prebuilt Docker image, it is
> >> less
> >> > clear to me if that is fine or not. In the interests of resolving this
> >> > question one way or the other, I opened a question asking about this
> >> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
> >> >
> >> > About Dylan's questions: my feeling is that we should go ahead and
> >> enable
> >> > automated pushes to Docker Hub, and provide some appropriate language
> >> > around what people should expect out of it. I don't think
> >> 'experimental' is
> >> > the right word, but we should be clear around exactly what contract we
> >> are
> >> > adhering to. Is it something people can expect to be published with
> each
> >> > release? Is it something that we are going to build and test as part
> of
> >> the
> >> > release process, or are we going to publish it via automation without
> >> any
> >> > testing? Is it something we expect people to use in production, or
> >> > something we only expect people to use for evaluation? If it is
> >> something
> >> > we expect people to use in production, do we expect them to use it in
> >> any
> >> > particular way? Will we be guaranteeing certain things (file layout,
> >> etc)
> >> > that provide a stable interface for people to build derived images
> from?
> >> >
> >> > The path of least resistance to answering these questions is to say
> that
> >> > the Docker image is provided in the hopes that it is useful, but that
> >> it is
> >> > done via an automated build, without any pre-release testing, and
> >> without
> >> > any particular guarantees about the 'interface' it provides. If this
> is
> >> the
> >> > case then I would suggest putting it up on Docker Hub with an
> >> appropriate
> >> > disclaimer and not promoting it too much. (We might very well end up
> >> > pushing images every once in a while that don't work right, and it
> would
> >> > reflect poorly on the project to have those be prominently linked-to.)
> >> It
> >> > becomes easier to strengthen these guarantees if we add an automated
> >> test
> >> > suite that we can run before releases which verifies functionality and
> >> > interface adherence.
> >> >
> >> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
> >> <rm...@vmware.com.invalid>
> >> > wrote:
> >> >
> >> > > This is purely a packaging exercise. I don't see a reason to mark
> >> this as
> >> > > experimental.
> >> > >
> >> > > Rajiv.
> >> > > ________________________________
> >> > > From: Dylan Wylie <dy...@apache.org>
> >> > > Sent: Friday, February 8, 2019 6:08:47 AM
> >> > > To: dev@druid.apache.org
> >> > > Subject: Re: docker build
> >> > >
> >> > > I believe all we have to do is submit a ticket to Apache's
> >> Infrastructure
> >> > > team and then we'll have some automatic process that'll
> automatically
> >> > > update docker-hub with images relating to each release.
> >> > >
> >> > > I guess there's two open questions I think we should reach a
> >> consensus on
> >> > > (others feel free to add more!).
> >> > >
> >> > > - Are we as a community happy to "support" an additional release
> >> > artefact?
> >> > > I'm happy to try to incorporate this into my employer's testing
> >> > > infrastructure to help catch any regressions on future releases but
> >> > that's
> >> > > just one data point on each release.
> >> > >
> >> > > - Along the same vein, do we follow the same process as we do with
> new
> >> > > features and mark this as experimental for some time?
> >> > >
> >> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> >> > >
> >> > > > Now that
> >> > >
> >> >
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> >> > > is merged
> >> > > > (thank you!)
> >> > > >
> >> > > > who can get this set to build into Dockerhub? Presumably
> >> automatically
> >> > > on a
> >> > > > 'tag' of the repo.
> >> > > >
> >> > > > Once that is done it is much more convenient for folks to use this
> >> > tool.
> >> > > >
> >> > > > --don
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
A discussion is progressing on
https://issues.apache.org/jira/browse/LEGAL-437. It doesn't seem to have
got anywhere firm yet.
On Fri, Feb 8, 2019 at 12:23 PM Gian Merlino <gi...@apache.org> wrote:
> I don't think anything is strictly needed from you at this point, but
> things happen when people drive them, and participation in that effort
> would help make sure it gets done. I think at this point the tasks on our
> end are watching LEGAL-437 for advice (or making it moot by removing the
> MySQL jar), asking Infra to set up automated builds once that is sorted
> out, and building some kind of consensus around how we'll label and promote
> the Docker images.
>
> On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
>
>> i'd be fine w/ removing the mysql, i'm using postgresql for the metadata.
>> if this is the case we should consider relfecting postgres as the default
>> metadata in the docs.
>> however, i think this is mere aggregation under the gpl license, and the
>> docker image tends to have other (e.g. bash) gpl code. druid's start
>> scripts are all bash-specific as an example.
>>
>> I'm not clear if anything further is needed of me, i'm hoping to get an
>> automated build going into dockerhub, and tagged w/ each release. i think
>> this will help adoption.
>>
>>
>>
>> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
>>
>> > First off thanks a lot for your work here Don!!
>> >
>> > I really do think, though, that we need to be careful about the
>> inclusion
>> > of the MySQL connector jar. ASF legal has been clear in the past that
>> ASF
>> > projects should not distribute it as part of binary convenience
>> releases:
>> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
>> > Dockerfile in the repo is probably fine: in that case we are not
>> > distributing the jar itself, just, essentially, a pointer to how to
>> > download it. But if we start offering a prebuilt Docker image, it is
>> less
>> > clear to me if that is fine or not. In the interests of resolving this
>> > question one way or the other, I opened a question asking about this
>> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
>> >
>> > About Dylan's questions: my feeling is that we should go ahead and
>> enable
>> > automated pushes to Docker Hub, and provide some appropriate language
>> > around what people should expect out of it. I don't think
>> 'experimental' is
>> > the right word, but we should be clear around exactly what contract we
>> are
>> > adhering to. Is it something people can expect to be published with each
>> > release? Is it something that we are going to build and test as part of
>> the
>> > release process, or are we going to publish it via automation without
>> any
>> > testing? Is it something we expect people to use in production, or
>> > something we only expect people to use for evaluation? If it is
>> something
>> > we expect people to use in production, do we expect them to use it in
>> any
>> > particular way? Will we be guaranteeing certain things (file layout,
>> etc)
>> > that provide a stable interface for people to build derived images from?
>> >
>> > The path of least resistance to answering these questions is to say that
>> > the Docker image is provided in the hopes that it is useful, but that
>> it is
>> > done via an automated build, without any pre-release testing, and
>> without
>> > any particular guarantees about the 'interface' it provides. If this is
>> the
>> > case then I would suggest putting it up on Docker Hub with an
>> appropriate
>> > disclaimer and not promoting it too much. (We might very well end up
>> > pushing images every once in a while that don't work right, and it would
>> > reflect poorly on the project to have those be prominently linked-to.)
>> It
>> > becomes easier to strengthen these guarantees if we add an automated
>> test
>> > suite that we can run before releases which verifies functionality and
>> > interface adherence.
>> >
>> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani
>> <rm...@vmware.com.invalid>
>> > wrote:
>> >
>> > > This is purely a packaging exercise. I don't see a reason to mark
>> this as
>> > > experimental.
>> > >
>> > > Rajiv.
>> > > ________________________________
>> > > From: Dylan Wylie <dy...@apache.org>
>> > > Sent: Friday, February 8, 2019 6:08:47 AM
>> > > To: dev@druid.apache.org
>> > > Subject: Re: docker build
>> > >
>> > > I believe all we have to do is submit a ticket to Apache's
>> Infrastructure
>> > > team and then we'll have some automatic process that'll automatically
>> > > update docker-hub with images relating to each release.
>> > >
>> > > I guess there's two open questions I think we should reach a
>> consensus on
>> > > (others feel free to add more!).
>> > >
>> > > - Are we as a community happy to "support" an additional release
>> > artefact?
>> > > I'm happy to try to incorporate this into my employer's testing
>> > > infrastructure to help catch any regressions on future releases but
>> > that's
>> > > just one data point on each release.
>> > >
>> > > - Along the same vein, do we follow the same process as we do with new
>> > > features and mark this as experimental for some time?
>> > >
>> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
>> > >
>> > > > Now that
>> > >
>> >
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
>> > > is merged
>> > > > (thank you!)
>> > > >
>> > > > who can get this set to build into Dockerhub? Presumably
>> automatically
>> > > on a
>> > > > 'tag' of the repo.
>> > > >
>> > > > Once that is done it is much more convenient for folks to use this
>> > tool.
>> > > >
>> > > > --don
>> > > >
>> > >
>> >
>>
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
I don't think anything is strictly needed from you at this point, but
things happen when people drive them, and participation in that effort
would help make sure it gets done. I think at this point the tasks on our
end are watching LEGAL-437 for advice (or making it moot by removing the
MySQL jar), asking Infra to set up automated builds once that is sorted
out, and building some kind of consensus around how we'll label and promote
the Docker images.
On Fri, Feb 8, 2019 at 12:13 PM Don Bowman <do...@agilicus.com> wrote:
> i'd be fine w/ removing the mysql, i'm using postgresql for the metadata.
> if this is the case we should consider relfecting postgres as the default
> metadata in the docs.
> however, i think this is mere aggregation under the gpl license, and the
> docker image tends to have other (e.g. bash) gpl code. druid's start
> scripts are all bash-specific as an example.
>
> I'm not clear if anything further is needed of me, i'm hoping to get an
> automated build going into dockerhub, and tagged w/ each release. i think
> this will help adoption.
>
>
>
> On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
>
> > First off thanks a lot for your work here Don!!
> >
> > I really do think, though, that we need to be careful about the inclusion
> > of the MySQL connector jar. ASF legal has been clear in the past that ASF
> > projects should not distribute it as part of binary convenience releases:
> > https://issues.apache.org/jira/browse/LEGAL-200. I think having the
> > Dockerfile in the repo is probably fine: in that case we are not
> > distributing the jar itself, just, essentially, a pointer to how to
> > download it. But if we start offering a prebuilt Docker image, it is less
> > clear to me if that is fine or not. In the interests of resolving this
> > question one way or the other, I opened a question asking about this
> > specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
> >
> > About Dylan's questions: my feeling is that we should go ahead and enable
> > automated pushes to Docker Hub, and provide some appropriate language
> > around what people should expect out of it. I don't think 'experimental'
> is
> > the right word, but we should be clear around exactly what contract we
> are
> > adhering to. Is it something people can expect to be published with each
> > release? Is it something that we are going to build and test as part of
> the
> > release process, or are we going to publish it via automation without any
> > testing? Is it something we expect people to use in production, or
> > something we only expect people to use for evaluation? If it is something
> > we expect people to use in production, do we expect them to use it in any
> > particular way? Will we be guaranteeing certain things (file layout, etc)
> > that provide a stable interface for people to build derived images from?
> >
> > The path of least resistance to answering these questions is to say that
> > the Docker image is provided in the hopes that it is useful, but that it
> is
> > done via an automated build, without any pre-release testing, and without
> > any particular guarantees about the 'interface' it provides. If this is
> the
> > case then I would suggest putting it up on Docker Hub with an appropriate
> > disclaimer and not promoting it too much. (We might very well end up
> > pushing images every once in a while that don't work right, and it would
> > reflect poorly on the project to have those be prominently linked-to.) It
> > becomes easier to strengthen these guarantees if we add an automated test
> > suite that we can run before releases which verifies functionality and
> > interface adherence.
> >
> > On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani <rmordani@vmware.com.invalid
> >
> > wrote:
> >
> > > This is purely a packaging exercise. I don't see a reason to mark this
> as
> > > experimental.
> > >
> > > Rajiv.
> > > ________________________________
> > > From: Dylan Wylie <dy...@apache.org>
> > > Sent: Friday, February 8, 2019 6:08:47 AM
> > > To: dev@druid.apache.org
> > > Subject: Re: docker build
> > >
> > > I believe all we have to do is submit a ticket to Apache's
> Infrastructure
> > > team and then we'll have some automatic process that'll automatically
> > > update docker-hub with images relating to each release.
> > >
> > > I guess there's two open questions I think we should reach a consensus
> on
> > > (others feel free to add more!).
> > >
> > > - Are we as a community happy to "support" an additional release
> > artefact?
> > > I'm happy to try to incorporate this into my employer's testing
> > > infrastructure to help catch any regressions on future releases but
> > that's
> > > just one data point on each release.
> > >
> > > - Along the same vein, do we follow the same process as we do with new
> > > features and mark this as experimental for some time?
> > >
> > > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> > >
> > > > Now that
> > >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> > > is merged
> > > > (thank you!)
> > > >
> > > > who can get this set to build into Dockerhub? Presumably
> automatically
> > > on a
> > > > 'tag' of the repo.
> > > >
> > > > Once that is done it is much more convenient for folks to use this
> > tool.
> > > >
> > > > --don
> > > >
> > >
> >
>
Re: docker build
Posted by Don Bowman <do...@agilicus.com>.
i'd be fine w/ removing the mysql, i'm using postgresql for the metadata.
if this is the case we should consider relfecting postgres as the default
metadata in the docs.
however, i think this is mere aggregation under the gpl license, and the
docker image tends to have other (e.g. bash) gpl code. druid's start
scripts are all bash-specific as an example.
I'm not clear if anything further is needed of me, i'm hoping to get an
automated build going into dockerhub, and tagged w/ each release. i think
this will help adoption.
On Fri, 8 Feb 2019 at 14:22, Gian Merlino <gi...@apache.org> wrote:
> First off thanks a lot for your work here Don!!
>
> I really do think, though, that we need to be careful about the inclusion
> of the MySQL connector jar. ASF legal has been clear in the past that ASF
> projects should not distribute it as part of binary convenience releases:
> https://issues.apache.org/jira/browse/LEGAL-200. I think having the
> Dockerfile in the repo is probably fine: in that case we are not
> distributing the jar itself, just, essentially, a pointer to how to
> download it. But if we start offering a prebuilt Docker image, it is less
> clear to me if that is fine or not. In the interests of resolving this
> question one way or the other, I opened a question asking about this
> specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
>
> About Dylan's questions: my feeling is that we should go ahead and enable
> automated pushes to Docker Hub, and provide some appropriate language
> around what people should expect out of it. I don't think 'experimental' is
> the right word, but we should be clear around exactly what contract we are
> adhering to. Is it something people can expect to be published with each
> release? Is it something that we are going to build and test as part of the
> release process, or are we going to publish it via automation without any
> testing? Is it something we expect people to use in production, or
> something we only expect people to use for evaluation? If it is something
> we expect people to use in production, do we expect them to use it in any
> particular way? Will we be guaranteeing certain things (file layout, etc)
> that provide a stable interface for people to build derived images from?
>
> The path of least resistance to answering these questions is to say that
> the Docker image is provided in the hopes that it is useful, but that it is
> done via an automated build, without any pre-release testing, and without
> any particular guarantees about the 'interface' it provides. If this is the
> case then I would suggest putting it up on Docker Hub with an appropriate
> disclaimer and not promoting it too much. (We might very well end up
> pushing images every once in a while that don't work right, and it would
> reflect poorly on the project to have those be prominently linked-to.) It
> becomes easier to strengthen these guarantees if we add an automated test
> suite that we can run before releases which verifies functionality and
> interface adherence.
>
> On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani <rm...@vmware.com.invalid>
> wrote:
>
> > This is purely a packaging exercise. I don't see a reason to mark this as
> > experimental.
> >
> > Rajiv.
> > ________________________________
> > From: Dylan Wylie <dy...@apache.org>
> > Sent: Friday, February 8, 2019 6:08:47 AM
> > To: dev@druid.apache.org
> > Subject: Re: docker build
> >
> > I believe all we have to do is submit a ticket to Apache's Infrastructure
> > team and then we'll have some automatic process that'll automatically
> > update docker-hub with images relating to each release.
> >
> > I guess there's two open questions I think we should reach a consensus on
> > (others feel free to add more!).
> >
> > - Are we as a community happy to "support" an additional release
> artefact?
> > I'm happy to try to incorporate this into my employer's testing
> > infrastructure to help catch any regressions on future releases but
> that's
> > just one data point on each release.
> >
> > - Along the same vein, do we follow the same process as we do with new
> > features and mark this as experimental for some time?
> >
> > On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> >
> > > Now that
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> > is merged
> > > (thank you!)
> > >
> > > who can get this set to build into Dockerhub? Presumably automatically
> > on a
> > > 'tag' of the repo.
> > >
> > > Once that is done it is much more convenient for folks to use this
> tool.
> > >
> > > --don
> > >
> >
>
Re: docker build
Posted by Gian Merlino <gi...@apache.org>.
First off thanks a lot for your work here Don!!
I really do think, though, that we need to be careful about the inclusion
of the MySQL connector jar. ASF legal has been clear in the past that ASF
projects should not distribute it as part of binary convenience releases:
https://issues.apache.org/jira/browse/LEGAL-200. I think having the
Dockerfile in the repo is probably fine: in that case we are not
distributing the jar itself, just, essentially, a pointer to how to
download it. But if we start offering a prebuilt Docker image, it is less
clear to me if that is fine or not. In the interests of resolving this
question one way or the other, I opened a question asking about this
specific situation: https://issues.apache.org/jira/browse/LEGAL-437.
About Dylan's questions: my feeling is that we should go ahead and enable
automated pushes to Docker Hub, and provide some appropriate language
around what people should expect out of it. I don't think 'experimental' is
the right word, but we should be clear around exactly what contract we are
adhering to. Is it something people can expect to be published with each
release? Is it something that we are going to build and test as part of the
release process, or are we going to publish it via automation without any
testing? Is it something we expect people to use in production, or
something we only expect people to use for evaluation? If it is something
we expect people to use in production, do we expect them to use it in any
particular way? Will we be guaranteeing certain things (file layout, etc)
that provide a stable interface for people to build derived images from?
The path of least resistance to answering these questions is to say that
the Docker image is provided in the hopes that it is useful, but that it is
done via an automated build, without any pre-release testing, and without
any particular guarantees about the 'interface' it provides. If this is the
case then I would suggest putting it up on Docker Hub with an appropriate
disclaimer and not promoting it too much. (We might very well end up
pushing images every once in a while that don't work right, and it would
reflect poorly on the project to have those be prominently linked-to.) It
becomes easier to strengthen these guarantees if we add an automated test
suite that we can run before releases which verifies functionality and
interface adherence.
On Fri, Feb 8, 2019 at 7:14 AM Rajiv Mordani <rm...@vmware.com.invalid>
wrote:
> This is purely a packaging exercise. I don't see a reason to mark this as
> experimental.
>
> Rajiv.
> ________________________________
> From: Dylan Wylie <dy...@apache.org>
> Sent: Friday, February 8, 2019 6:08:47 AM
> To: dev@druid.apache.org
> Subject: Re: docker build
>
> I believe all we have to do is submit a ticket to Apache's Infrastructure
> team and then we'll have some automatic process that'll automatically
> update docker-hub with images relating to each release.
>
> I guess there's two open questions I think we should reach a consensus on
> (others feel free to add more!).
>
> - Are we as a community happy to "support" an additional release artefact?
> I'm happy to try to incorporate this into my employer's testing
> infrastructure to help catch any regressions on future releases but that's
> just one data point on each release.
>
> - Along the same vein, do we follow the same process as we do with new
> features and mark this as experimental for some time?
>
> On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
>
> > Now that
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0
> is merged
> > (thank you!)
> >
> > who can get this set to build into Dockerhub? Presumably automatically
> on a
> > 'tag' of the repo.
> >
> > Once that is done it is much more convenient for folks to use this tool.
> >
> > --don
> >
>
Re: docker build
Posted by Rajiv Mordani <rm...@vmware.com.INVALID>.
This is purely a packaging exercise. I don't see a reason to mark this as experimental.
Rajiv.
________________________________
From: Dylan Wylie <dy...@apache.org>
Sent: Friday, February 8, 2019 6:08:47 AM
To: dev@druid.apache.org
Subject: Re: docker build
I believe all we have to do is submit a ticket to Apache's Infrastructure
team and then we'll have some automatic process that'll automatically
update docker-hub with images relating to each release.
I guess there's two open questions I think we should reach a consensus on
(others feel free to add more!).
- Are we as a community happy to "support" an additional release artefact?
I'm happy to try to incorporate this into my employer's testing
infrastructure to help catch any regressions on future releases but that's
just one data point on each release.
- Along the same vein, do we follow the same process as we do with new
features and mark this as experimental for some time?
On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> Now that https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6896&data=02%7C01%7Crmordani%40vmware.com%7C942b2af1dfb740fcbed308d68dcef937%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C636852317419449405&sdata=EXigZIBkKiatM0rEgyQRoxA9ER8u8amiAfPN0MghzjE%3D&reserved=0 is merged
> (thank you!)
>
> who can get this set to build into Dockerhub? Presumably automatically on a
> 'tag' of the repo.
>
> Once that is done it is much more convenient for folks to use this tool.
>
> --don
>
Re: docker build
Posted by Dylan Wylie <dy...@apache.org>.
I believe all we have to do is submit a ticket to Apache's Infrastructure
team and then we'll have some automatic process that'll automatically
update docker-hub with images relating to each release.
I guess there's two open questions I think we should reach a consensus on
(others feel free to add more!).
- Are we as a community happy to "support" an additional release artefact?
I'm happy to try to incorporate this into my employer's testing
infrastructure to help catch any regressions on future releases but that's
just one data point on each release.
- Along the same vein, do we follow the same process as we do with new
features and mark this as experimental for some time?
On Fri, 8 Feb 2019 at 13:25, Don Bowman <do...@agilicus.com> wrote:
> Now that https://github.com/apache/incubator-druid/pull/6896 is merged
> (thank you!)
>
> who can get this set to build into Dockerhub? Presumably automatically on a
> 'tag' of the repo.
>
> Once that is done it is much more convenient for folks to use this tool.
>
> --don
>