You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Tom Barber <to...@spicule.co.uk> on 2018/10/14 10:10:37 UTC
OODT docker builds
I’m interested in the Dockerization of OODT but also conscious that most
people use RADIX to build their stuff, which make’s overriding bits of a
prebuilt image tricky.
I’m wondering if its worth adding an optional Docker profile to RADIX to
add a Docker build step to the backend of people’s RADIX builds if they so
wanted.
Thoughts?
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Tom Barber <to...@spicule.co.uk>.
Also don’t forget, Docker containers, don’t ship Kernel’s etc and all the
system stuff, which is what keeps them lightweight.
On 15 October 2018 at 17:43:37, Tom Barber (tom@spicule.co.uk) wrote:
Not at all and if you want to run OODT on Kubernetes for example, that
would be how you’d do it, that way you can upgrade, scale, restart and fail
components without the entire stack falling over.
In terms of disk space, don’t forget each image is built on layers, so for
example, openjdk-8 on alpine is 56mb, that layer would then be used across
all base images so its only installed once on each host, then you’ve got
your file manager for example which would check in currently at 62MB, so
the entire image size would be 118MB which you could then deploy on 1 node,
or 100 nodes.
Then say you’ve got opsui as another dependency, that would be
tomcat:8 (463mb)+ opsui(73mb)==536mb
But say you have no interest in workflow etc, thats all you’d deploy.
In reality it would be much more flexible and much more inline with how
docker containers should be deployed, which is as a single process
container not as a bunch of processes all stuck into 1 unit.
On 15 October 2018 at 17:32:32, Chris Mattmann (mattmann@apache.org) wrote:
Isn’t an image per component really heavyweight?
*From:* Tom Barber <to...@spicule.co.uk>
*Date:* Monday, October 15, 2018 at 8:26 AM
*To:* Imesha Sudasingha <im...@cse.mrt.ac.lk>
*Cc:* dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
*Subject:* Re: OODT docker builds
Why aren’t we doing so?! :)
Lack of cycles and young kids ;)
I’ll take a stab at it and see where we get to outside of RADIX to get the
stack in distinct containers and then we’ll look at integrating it into the
main build then.
Tom
On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Yes. Agree with you. That is something I have been planning to ask you for
long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested
we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in
the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
I was thinking about the outputs. Currently everything is 1 docker file
which is fine for some deployments and not for others.
In the RADIX build we should also build individual containers for each
component. For the deployment we could also have a docker-compose file and
a K8S Helm setup so that you could deploy a distributed OODT setup from
your RADIX output, this could also have the ZK stuff in it so we can
properly utilise the distributed nature we started constructing with the FM
ZK changes.
On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> I’m interested in the Dockerization of OODT but also conscious that most
> people use RADIX to build their stuff, which make’s overriding bits of a
> prebuilt image tricky.
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
> add a Docker build step to the backend of people’s RADIX builds if they so
> wanted.
>
> Thoughts?
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Imesha Sudasingha <im...@cse.mrt.ac.lk>.
Nice explanation Tom.
In addition to that, we will be able to run these containers in multiple
pods within
a kubernetes cluster when being used for an actual use case (production).
This way, it will be easy to deploy, scale and manage.
Under those conditions, we may be able to run one component per node as
well with Kube-DNS is there
to avoid hardcoding IPs and allow service discovery without a problem.
On Mon, 15 Oct 2018 at 23:55, Tom Barber <to...@spicule.co.uk> wrote:
> internally as they only 1 process they should run with very little
> overhead.
>
>
> I can vouch also though that IO certainly sucks really really badly on my
> Mac compared to linux box, and if SK reads this I'm sure he can also relay
> some docker horror stories on Mac.
>
> On Mac it's better since they got rid of the virtualbox bridge, but it's
> still pretty bad.
>
> Tom
>
> On Mon, 15 Oct 2018, 19:20 Chris Mattmann, <ma...@apache.org> wrote:
>
> > Interesting OK. B/c the more containers for me on my local laptop
> > typically eat up way more memory and CPU…but maybe that’s just
> > me on a Mac lol
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: Tom Barber <to...@spicule.co.uk>
> > Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Date: Monday, October 15, 2018 at 9:44 AM
> > To: Imesha Sudasingha <im...@cse.mrt.ac.lk>, Chris Mattmann <
> > mattmann@apache.org>
> > Cc: dev <de...@oodt.apache.org>
> > Subject: Re: OODT docker builds
> >
> >
> >
> > Not at all and if you want to run OODT on Kubernetes for example, that
> >
> > would be how you’d do it, that way you can upgrade, scale, restart and
> fail
> >
> > components without the entire stack falling over.
> >
> >
> >
> > In terms of disk space, don’t forget each image is built on layers, so
> for
> >
> > example, openjdk-8 on alpine is 56mb, that layer would then be used
> across
> >
> > all base images so its only installed once on each host, then you’ve got
> >
> > your file manager for example which would check in currently at 62MB, so
> >
> > the entire image size would be 118MB which you could then deploy on 1
> node,
> >
> > or 100 nodes.
> >
> >
> >
> > Then say you’ve got opsui as another dependency, that would be
> >
> >
> >
> > tomcat:8 (463mb)+ opsui(73mb)==536mb
> >
> >
> >
> > But say you have no interest in workflow etc, thats all you’d deploy.
> >
> >
> >
> > In reality it would be much more flexible and much more inline with how
> >
> > docker containers should be deployed, which is as a single process
> >
> > container not as a bunch of processes all stuck into 1 unit.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 15 October 2018 at 17:32:32, Chris Mattmann (mattmann@apache.org)
> > wrote:
> >
> >
> >
> > Isn’t an image per component really heavyweight?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *From: *Tom Barber <to...@spicule.co.uk>
> >
> > *Date: *Monday, October 15, 2018 at 8:26 AM
> >
> > *To: *Imesha Sudasingha <im...@cse.mrt.ac.lk>
> >
> > *Cc: *dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
> >
> > *Subject: *Re: OODT docker builds
> >
> >
> >
> >
> >
> >
> >
> > Why aren’t we doing so?! :)
> >
> >
> >
> >
> >
> >
> >
> > Lack of cycles and young kids ;)
> >
> >
> >
> >
> >
> >
> >
> > I’ll take a stab at it and see where we get to outside of RADIX to get
> the
> >
> > stack in distinct containers and then we’ll look at integrating it into
> the
> >
> > main build then.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Tom
> >
> >
> >
> >
> >
> >
> >
> > On 15 October 2018 at 13:07:56, Imesha Sudasingha (
> imesha.13@cse.mrt.ac.lk
> > )
> >
> > wrote:
> >
> >
> >
> > Yes. Agree with you. That is something I have been planning to ask you
> for
> >
> > long; why aren't we doing so.
> >
> >
> >
> > I like the idea of having a docker image per component and as you
> suggested
> >
> > we can create docker-compose
> >
> >
> >
> > or kubernetes setup for deployments. I like that direction ;-)
> >
> >
> >
> >
> >
> >
> >
> > As an starting point, we can add an all-in-one docker image to be built
> in
> >
> > the RADIX build, right?
> >
> >
> >
> > If you start off, I will be able to join you along the way.
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Imesha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
> >
> >
> >
> > I was thinking about the outputs. Currently everything is 1 docker file
> >
> > which is fine for some deployments and not for others.
> >
> >
> >
> >
> >
> >
> >
> > In the RADIX build we should also build individual containers for each
> >
> > component. For the deployment we could also have a docker-compose file
> and
> >
> > a K8S Helm setup so that you could deploy a distributed OODT setup from
> >
> > your RADIX output, this could also have the ZK stuff in it so we can
> >
> > properly utilise the distributed nature we started constructing with the
> FM
> >
> > ZK changes.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 15 October 2018 at 05:08:41, Imesha Sudasingha (
> imesha.13@cse.mrt.ac.lk
> > )
> >
> > wrote:
> >
> >
> >
> > Hi Tom,
> >
> >
> >
> > I think this will be great since people are adopting docker more and more
> >
> > and even for a user
> >
> > once they have built a customized docker image, they can share it among
> the
> >
> > peers
> >
> > reducing the time spent for configuration by each individual.
> >
> >
> >
> > Also we have another option ;-) With distributed configuration management
> >
> > which I implemented,
> >
> > users can ask OODT components to download configuration published in
> >
> > zookeeper. But this will
> >
> > require zookeeper to be running (either as a container or standalone). As
> >
> > per my understanding,
> >
> > configuration is the problem we need to solve when using a pre-built
> docker
> >
> > image?
> >
> >
> >
> > In future, if we are able to implement
> >
> > https://issues.apache.org/jira/browse/OODT-977
> >
> > we will be able to run multiple file managers in multiple containers
> which
> >
> > will allow
> >
> > the load to be distributed and query all at once. So, docker will be the
> >
> > way to go as I see it.
> >
> > What do you think?
> >
> >
> >
> > Thanks,
> >
> > Imesha
> >
> >
> >
> >
> >
> > On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> >
> >
> >
> > I’m interested in the Dockerization of OODT but also conscious that most
> >
> > people use RADIX to build their stuff, which make’s overriding bits of a
> >
> > prebuilt image tricky.
> >
> >
> >
> > I’m wondering if its worth adding an optional Docker profile to RADIX to
> >
> > add a Docker build step to the backend of people’s RADIX builds if they
> so
> >
> > wanted.
> >
> >
> >
> > Thoughts?
> >
> >
> >
> > --
> >
> >
> >
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> >
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> >
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > All engagements
> >
> > are subject to Spicule Terms and Conditions of Business. This email and
> >
> > its
> >
> > contents are intended solely for the individual to whom it is addressed
> >
> > and
> >
> > may contain information that is confidential, privileged or otherwise
> >
> > protected from disclosure, distributing or copying. Any views or opinions
> >
> > presented in this email are solely those of the author and do not
> >
> > necessarily represent those of Spicule Limited. The company accepts no
> >
> > liability for any damage caused by any virus transmitted by this email.
> If
> >
> > you have received this message in error, please notify us immediately by
> >
> > reply email before deleting it from your system. Service of legal notice
> >
> > cannot be effected on Spicule Limited by email.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> 09954122.
> >
> > Registered office: First Floor, Telecom House, 125-135 Preston Road,
> >
> > Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> >
> >
> >
> > All engagements are subject to Spicule Terms and Conditions of Business.
> >
> > This email and its contents are intended solely for the individual to
> whom
> >
> > it is addressed and may contain information that is confidential,
> >
> > privileged or otherwise protected from disclosure, distributing or
> copying.
> >
> > Any views or opinions presented in this email are solely those of the
> >
> > author and do not necessarily represent those of Spicule Limited. The
> >
> > company accepts no liability for any damage caused by any virus
> transmitted
> >
> > by this email. If you have received this message in error, please notify
> us
> >
> > immediately by reply email before deleting it from your system. Service
> of
> >
> > legal notice cannot be effected on Spicule Limited by email.
> >
> >
> >
> >
> >
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> 09954122.
> >
> > Registered office: First Floor, Telecom House, 125-135 Preston Road,
> >
> > Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> >
> >
> >
> > All engagements are subject to Spicule Terms and Conditions of Business.
> >
> > This email and its contents are intended solely for the individual to
> whom
> >
> > it is addressed and may contain information that is confidential,
> >
> > privileged or otherwise protected from disclosure, distributing or
> copying.
> >
> > Any views or opinions presented in this email are solely those of the
> >
> > author and do not necessarily represent those of Spicule Limited. The
> >
> > company accepts no liability for any damage caused by any virus
> transmitted
> >
> > by this email. If you have received this message in error, please notify
> us
> >
> > immediately by reply email before deleting it from your system. Service
> of
> >
> > legal notice cannot be effected on Spicule Limited by email.
> >
> >
> >
> > --
> >
> >
> >
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> >
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> >
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > All engagements
> >
> > are subject to Spicule Terms and Conditions of Business. This email and
> > its
> >
> > contents are intended solely for the individual to whom it is addressed
> > and
> >
> > may contain information that is confidential, privileged or otherwise
> >
> > protected from disclosure, distributing or copying. Any views or opinions
> >
> > presented in this email are solely those of the author and do not
> >
> > necessarily represent those of Spicule Limited. The company accepts no
> >
> > liability for any damage caused by any virus transmitted by this email.
> If
> >
> > you have received this message in error, please notify us immediately by
> >
> > reply email before deleting it from your system. Service of legal notice
> >
> > cannot be effected on Spicule Limited by email.
> >
> >
> >
> >
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
Re: OODT docker builds
Posted by Tom Barber <to...@spicule.co.uk>.
internally as they only 1 process they should run with very little
overhead.
I can vouch also though that IO certainly sucks really really badly on my
Mac compared to linux box, and if SK reads this I'm sure he can also relay
some docker horror stories on Mac.
On Mac it's better since they got rid of the virtualbox bridge, but it's
still pretty bad.
Tom
On Mon, 15 Oct 2018, 19:20 Chris Mattmann, <ma...@apache.org> wrote:
> Interesting OK. B/c the more containers for me on my local laptop
> typically eat up way more memory and CPU…but maybe that’s just
> me on a Mac lol
>
>
>
>
>
>
>
>
>
> From: Tom Barber <to...@spicule.co.uk>
> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Date: Monday, October 15, 2018 at 9:44 AM
> To: Imesha Sudasingha <im...@cse.mrt.ac.lk>, Chris Mattmann <
> mattmann@apache.org>
> Cc: dev <de...@oodt.apache.org>
> Subject: Re: OODT docker builds
>
>
>
> Not at all and if you want to run OODT on Kubernetes for example, that
>
> would be how you’d do it, that way you can upgrade, scale, restart and fail
>
> components without the entire stack falling over.
>
>
>
> In terms of disk space, don’t forget each image is built on layers, so for
>
> example, openjdk-8 on alpine is 56mb, that layer would then be used across
>
> all base images so its only installed once on each host, then you’ve got
>
> your file manager for example which would check in currently at 62MB, so
>
> the entire image size would be 118MB which you could then deploy on 1 node,
>
> or 100 nodes.
>
>
>
> Then say you’ve got opsui as another dependency, that would be
>
>
>
> tomcat:8 (463mb)+ opsui(73mb)==536mb
>
>
>
> But say you have no interest in workflow etc, thats all you’d deploy.
>
>
>
> In reality it would be much more flexible and much more inline with how
>
> docker containers should be deployed, which is as a single process
>
> container not as a bunch of processes all stuck into 1 unit.
>
>
>
>
>
>
>
>
>
>
>
> On 15 October 2018 at 17:32:32, Chris Mattmann (mattmann@apache.org)
> wrote:
>
>
>
> Isn’t an image per component really heavyweight?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From: *Tom Barber <to...@spicule.co.uk>
>
> *Date: *Monday, October 15, 2018 at 8:26 AM
>
> *To: *Imesha Sudasingha <im...@cse.mrt.ac.lk>
>
> *Cc: *dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
>
> *Subject: *Re: OODT docker builds
>
>
>
>
>
>
>
> Why aren’t we doing so?! :)
>
>
>
>
>
>
>
> Lack of cycles and young kids ;)
>
>
>
>
>
>
>
> I’ll take a stab at it and see where we get to outside of RADIX to get the
>
> stack in distinct containers and then we’ll look at integrating it into the
>
> main build then.
>
>
>
>
>
>
>
>
>
>
>
> Tom
>
>
>
>
>
>
>
> On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk
> )
>
> wrote:
>
>
>
> Yes. Agree with you. That is something I have been planning to ask you for
>
> long; why aren't we doing so.
>
>
>
> I like the idea of having a docker image per component and as you suggested
>
> we can create docker-compose
>
>
>
> or kubernetes setup for deployments. I like that direction ;-)
>
>
>
>
>
>
>
> As an starting point, we can add an all-in-one docker image to be built in
>
> the RADIX build, right?
>
>
>
> If you start off, I will be able to join you along the way.
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Imesha
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
>
>
>
> I was thinking about the outputs. Currently everything is 1 docker file
>
> which is fine for some deployments and not for others.
>
>
>
>
>
>
>
> In the RADIX build we should also build individual containers for each
>
> component. For the deployment we could also have a docker-compose file and
>
> a K8S Helm setup so that you could deploy a distributed OODT setup from
>
> your RADIX output, this could also have the ZK stuff in it so we can
>
> properly utilise the distributed nature we started constructing with the FM
>
> ZK changes.
>
>
>
>
>
>
>
>
>
>
>
> On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk
> )
>
> wrote:
>
>
>
> Hi Tom,
>
>
>
> I think this will be great since people are adopting docker more and more
>
> and even for a user
>
> once they have built a customized docker image, they can share it among the
>
> peers
>
> reducing the time spent for configuration by each individual.
>
>
>
> Also we have another option ;-) With distributed configuration management
>
> which I implemented,
>
> users can ask OODT components to download configuration published in
>
> zookeeper. But this will
>
> require zookeeper to be running (either as a container or standalone). As
>
> per my understanding,
>
> configuration is the problem we need to solve when using a pre-built docker
>
> image?
>
>
>
> In future, if we are able to implement
>
> https://issues.apache.org/jira/browse/OODT-977
>
> we will be able to run multiple file managers in multiple containers which
>
> will allow
>
> the load to be distributed and query all at once. So, docker will be the
>
> way to go as I see it.
>
> What do you think?
>
>
>
> Thanks,
>
> Imesha
>
>
>
>
>
> On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
>
>
>
> I’m interested in the Dockerization of OODT but also conscious that most
>
> people use RADIX to build their stuff, which make’s overriding bits of a
>
> prebuilt image tricky.
>
>
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
>
> add a Docker build step to the backend of people’s RADIX builds if they so
>
> wanted.
>
>
>
> Thoughts?
>
>
>
> --
>
>
>
>
>
> Spicule Limited is registered in England & Wales. Company Number:
>
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
>
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
>
>
>
>
>
> All engagements
>
> are subject to Spicule Terms and Conditions of Business. This email and
>
> its
>
> contents are intended solely for the individual to whom it is addressed
>
> and
>
> may contain information that is confidential, privileged or otherwise
>
> protected from disclosure, distributing or copying. Any views or opinions
>
> presented in this email are solely those of the author and do not
>
> necessarily represent those of Spicule Limited. The company accepts no
>
> liability for any damage caused by any virus transmitted by this email. If
>
> you have received this message in error, please notify us immediately by
>
> reply email before deleting it from your system. Service of legal notice
>
> cannot be effected on Spicule Limited by email.
>
>
>
>
>
>
>
>
>
> Spicule Limited is registered in England & Wales. Company Number: 09954122.
>
> Registered office: First Floor, Telecom House, 125-135 Preston Road,
>
> Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
>
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
>
> This email and its contents are intended solely for the individual to whom
>
> it is addressed and may contain information that is confidential,
>
> privileged or otherwise protected from disclosure, distributing or copying.
>
> Any views or opinions presented in this email are solely those of the
>
> author and do not necessarily represent those of Spicule Limited. The
>
> company accepts no liability for any damage caused by any virus transmitted
>
> by this email. If you have received this message in error, please notify us
>
> immediately by reply email before deleting it from your system. Service of
>
> legal notice cannot be effected on Spicule Limited by email.
>
>
>
>
>
>
>
> Spicule Limited is registered in England & Wales. Company Number: 09954122.
>
> Registered office: First Floor, Telecom House, 125-135 Preston Road,
>
> Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
>
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
>
> This email and its contents are intended solely for the individual to whom
>
> it is addressed and may contain information that is confidential,
>
> privileged or otherwise protected from disclosure, distributing or copying.
>
> Any views or opinions presented in this email are solely those of the
>
> author and do not necessarily represent those of Spicule Limited. The
>
> company accepts no liability for any damage caused by any virus transmitted
>
> by this email. If you have received this message in error, please notify us
>
> immediately by reply email before deleting it from your system. Service of
>
> legal notice cannot be effected on Spicule Limited by email.
>
>
>
> --
>
>
>
>
>
> Spicule Limited is registered in England & Wales. Company Number:
>
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
>
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
>
>
>
>
>
> All engagements
>
> are subject to Spicule Terms and Conditions of Business. This email and
> its
>
> contents are intended solely for the individual to whom it is addressed
> and
>
> may contain information that is confidential, privileged or otherwise
>
> protected from disclosure, distributing or copying. Any views or opinions
>
> presented in this email are solely those of the author and do not
>
> necessarily represent those of Spicule Limited. The company accepts no
>
> liability for any damage caused by any virus transmitted by this email. If
>
> you have received this message in error, please notify us immediately by
>
> reply email before deleting it from your system. Service of legal notice
>
> cannot be effected on Spicule Limited by email.
>
>
>
>
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Chris Mattmann <ma...@apache.org>.
Interesting OK. B/c the more containers for me on my local laptop
typically eat up way more memory and CPU…but maybe that’s just
me on a Mac lol
From: Tom Barber <to...@spicule.co.uk>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Monday, October 15, 2018 at 9:44 AM
To: Imesha Sudasingha <im...@cse.mrt.ac.lk>, Chris Mattmann <ma...@apache.org>
Cc: dev <de...@oodt.apache.org>
Subject: Re: OODT docker builds
Not at all and if you want to run OODT on Kubernetes for example, that
would be how you’d do it, that way you can upgrade, scale, restart and fail
components without the entire stack falling over.
In terms of disk space, don’t forget each image is built on layers, so for
example, openjdk-8 on alpine is 56mb, that layer would then be used across
all base images so its only installed once on each host, then you’ve got
your file manager for example which would check in currently at 62MB, so
the entire image size would be 118MB which you could then deploy on 1 node,
or 100 nodes.
Then say you’ve got opsui as another dependency, that would be
tomcat:8 (463mb)+ opsui(73mb)==536mb
But say you have no interest in workflow etc, thats all you’d deploy.
In reality it would be much more flexible and much more inline with how
docker containers should be deployed, which is as a single process
container not as a bunch of processes all stuck into 1 unit.
On 15 October 2018 at 17:32:32, Chris Mattmann (mattmann@apache.org) wrote:
Isn’t an image per component really heavyweight?
*From: *Tom Barber <to...@spicule.co.uk>
*Date: *Monday, October 15, 2018 at 8:26 AM
*To: *Imesha Sudasingha <im...@cse.mrt.ac.lk>
*Cc: *dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
*Subject: *Re: OODT docker builds
Why aren’t we doing so?! :)
Lack of cycles and young kids ;)
I’ll take a stab at it and see where we get to outside of RADIX to get the
stack in distinct containers and then we’ll look at integrating it into the
main build then.
Tom
On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Yes. Agree with you. That is something I have been planning to ask you for
long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested
we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in
the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
I was thinking about the outputs. Currently everything is 1 docker file
which is fine for some deployments and not for others.
In the RADIX build we should also build individual containers for each
component. For the deployment we could also have a docker-compose file and
a K8S Helm setup so that you could deploy a distributed OODT setup from
your RADIX output, this could also have the ZK stuff in it so we can
properly utilise the distributed nature we started constructing with the FM
ZK changes.
On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
I’m interested in the Dockerization of OODT but also conscious that most
people use RADIX to build their stuff, which make’s overriding bits of a
prebuilt image tricky.
I’m wondering if its worth adding an optional Docker profile to RADIX to
add a Docker build step to the backend of people’s RADIX builds if they so
wanted.
Thoughts?
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and
its
contents are intended solely for the individual to whom it is addressed
and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Tom Barber <to...@spicule.co.uk>.
Not at all and if you want to run OODT on Kubernetes for example, that
would be how you’d do it, that way you can upgrade, scale, restart and fail
components without the entire stack falling over.
In terms of disk space, don’t forget each image is built on layers, so for
example, openjdk-8 on alpine is 56mb, that layer would then be used across
all base images so its only installed once on each host, then you’ve got
your file manager for example which would check in currently at 62MB, so
the entire image size would be 118MB which you could then deploy on 1 node,
or 100 nodes.
Then say you’ve got opsui as another dependency, that would be
tomcat:8 (463mb)+ opsui(73mb)==536mb
But say you have no interest in workflow etc, thats all you’d deploy.
In reality it would be much more flexible and much more inline with how
docker containers should be deployed, which is as a single process
container not as a bunch of processes all stuck into 1 unit.
On 15 October 2018 at 17:32:32, Chris Mattmann (mattmann@apache.org) wrote:
Isn’t an image per component really heavyweight?
*From: *Tom Barber <to...@spicule.co.uk>
*Date: *Monday, October 15, 2018 at 8:26 AM
*To: *Imesha Sudasingha <im...@cse.mrt.ac.lk>
*Cc: *dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
*Subject: *Re: OODT docker builds
Why aren’t we doing so?! :)
Lack of cycles and young kids ;)
I’ll take a stab at it and see where we get to outside of RADIX to get the
stack in distinct containers and then we’ll look at integrating it into the
main build then.
Tom
On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Yes. Agree with you. That is something I have been planning to ask you for
long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested
we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in
the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
I was thinking about the outputs. Currently everything is 1 docker file
which is fine for some deployments and not for others.
In the RADIX build we should also build individual containers for each
component. For the deployment we could also have a docker-compose file and
a K8S Helm setup so that you could deploy a distributed OODT setup from
your RADIX output, this could also have the ZK stuff in it so we can
properly utilise the distributed nature we started constructing with the FM
ZK changes.
On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> I’m interested in the Dockerization of OODT but also conscious that most
> people use RADIX to build their stuff, which make’s overriding bits of a
> prebuilt image tricky.
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
> add a Docker build step to the backend of people’s RADIX builds if they so
> wanted.
>
> Thoughts?
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
Spicule Limited is registered in England & Wales. Company Number: 09954122.
Registered office: First Floor, Telecom House, 125-135 Preston Road,
Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business.
This email and its contents are intended solely for the individual to whom
it is addressed and may contain information that is confidential,
privileged or otherwise protected from disclosure, distributing or copying.
Any views or opinions presented in this email are solely those of the
author and do not necessarily represent those of Spicule Limited. The
company accepts no liability for any damage caused by any virus transmitted
by this email. If you have received this message in error, please notify us
immediately by reply email before deleting it from your system. Service of
legal notice cannot be effected on Spicule Limited by email.
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Chris Mattmann <ma...@apache.org>.
Isn’t an image per component really heavyweight?
From: Tom Barber <to...@spicule.co.uk>
Date: Monday, October 15, 2018 at 8:26 AM
To: Imesha Sudasingha <im...@cse.mrt.ac.lk>
Cc: dev <de...@oodt.apache.org>, Chris Mattmann <ma...@apache.org>
Subject: Re: OODT docker builds
Why aren’t we doing so?! :)
Lack of cycles and young kids ;)
I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then.
Tom
On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk) wrote:
Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others.
In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes.
On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk) wrote:
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> I’m interested in the Dockerization of OODT but also conscious that most
> people use RADIX to build their stuff, which make’s overriding bits of a
> prebuilt image tricky.
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
> add a Docker build step to the backend of people’s RADIX builds if they so
> wanted.
>
> Thoughts?
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Tom Barber <to...@spicule.co.uk>.
Why aren’t we doing so?! :)
Lack of cycles and young kids ;)
I’ll take a stab at it and see where we get to outside of RADIX to get the
stack in distinct containers and then we’ll look at integrating it into the
main build then.
Tom
On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Yes. Agree with you. That is something I have been planning to ask you for
long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested
we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in
the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
> I was thinking about the outputs. Currently everything is 1 docker file
> which is fine for some deployments and not for others.
>
> In the RADIX build we should also build individual containers for each
> component. For the deployment we could also have a docker-compose file and
> a K8S Helm setup so that you could deploy a distributed OODT setup from
> your RADIX output, this could also have the ZK stuff in it so we can
> properly utilise the distributed nature we started constructing with the FM
> ZK changes.
>
>
> On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
> wrote:
>
> Hi Tom,
>
> I think this will be great since people are adopting docker more and more
> and even for a user
> once they have built a customized docker image, they can share it among the
> peers
> reducing the time spent for configuration by each individual.
>
> Also we have another option ;-) With distributed configuration management
> which I implemented,
> users can ask OODT components to download configuration published in
> zookeeper. But this will
> require zookeeper to be running (either as a container or standalone). As
> per my understanding,
> configuration is the problem we need to solve when using a pre-built docker
> image?
>
> In future, if we are able to implement
> https://issues.apache.org/jira/browse/OODT-977
> we will be able to run multiple file managers in multiple containers which
> will allow
> the load to be distributed and query all at once. So, docker will be the
> way to go as I see it.
> What do you think?
>
> Thanks,
> Imesha
>
>
> On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
>
> > I’m interested in the Dockerization of OODT but also conscious that most
> > people use RADIX to build their stuff, which make’s overriding bits of a
> > prebuilt image tricky.
> >
> > I’m wondering if its worth adding an optional Docker profile to RADIX to
> > add a Docker build step to the backend of people’s RADIX builds if they
> so
> > wanted.
> >
> > Thoughts?
> >
> > --
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> > All engagements
> > are subject to Spicule Terms and Conditions of Business. This email and
> > its
> > contents are intended solely for the individual to whom it is addressed
> > and
> > may contain information that is confidential, privileged or otherwise
> > protected from disclosure, distributing or copying. Any views or opinions
> > presented in this email are solely those of the author and do not
> > necessarily represent those of Spicule Limited. The company accepts no
> > liability for any damage caused by any virus transmitted by this email.
> If
> > you have received this message in error, please notify us immediately by
> > reply email before deleting it from your system. Service of legal notice
> > cannot be effected on Spicule Limited by email.
> >
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
> This email and its contents are intended solely for the individual to whom
> it is addressed and may contain information that is confidential,
> privileged or otherwise protected from disclosure, distributing or copying.
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of Spicule Limited. The
> company accepts no liability for any damage caused by any virus transmitted
> by this email. If you have received this message in error, please notify us
> immediately by reply email before deleting it from your system. Service of
> legal notice cannot be effected on Spicule Limited by email.
>
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Imesha Sudasingha <im...@cse.mrt.ac.lk>.
Yes. Agree with you. That is something I have been planning to ask you for
long; why aren't we doing so.
I like the idea of having a docker image per component and as you suggested
we can create docker-compose
or kubernetes setup for deployments. I like that direction ;-)
As an starting point, we can add an all-in-one docker image to be built in
the RADIX build, right?
If you start off, I will be able to join you along the way.
Thanks,
Imesha
On Mon, 15 Oct 2018 at 16:33, Tom Barber <to...@spicule.co.uk> wrote:
> I was thinking about the outputs. Currently everything is 1 docker file
> which is fine for some deployments and not for others.
>
> In the RADIX build we should also build individual containers for each
> component. For the deployment we could also have a docker-compose file and
> a K8S Helm setup so that you could deploy a distributed OODT setup from
> your RADIX output, this could also have the ZK stuff in it so we can
> properly utilise the distributed nature we started constructing with the FM
> ZK changes.
>
>
> On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
> wrote:
>
> Hi Tom,
>
> I think this will be great since people are adopting docker more and more
> and even for a user
> once they have built a customized docker image, they can share it among
> the
> peers
> reducing the time spent for configuration by each individual.
>
> Also we have another option ;-) With distributed configuration management
> which I implemented,
> users can ask OODT components to download configuration published in
> zookeeper. But this will
> require zookeeper to be running (either as a container or standalone). As
> per my understanding,
> configuration is the problem we need to solve when using a pre-built
> docker
> image?
>
> In future, if we are able to implement
> https://issues.apache.org/jira/browse/OODT-977
> we will be able to run multiple file managers in multiple containers which
> will allow
> the load to be distributed and query all at once. So, docker will be the
> way to go as I see it.
> What do you think?
>
> Thanks,
> Imesha
>
>
> On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
>
> > I’m interested in the Dockerization of OODT but also conscious that most
> > people use RADIX to build their stuff, which make’s overriding bits of a
> > prebuilt image tricky.
> >
> > I’m wondering if its worth adding an optional Docker profile to RADIX to
> > add a Docker build step to the backend of people’s RADIX builds if they
> so
> > wanted.
> >
> > Thoughts?
> >
> > --
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> > All engagements
> > are subject to Spicule Terms and Conditions of Business. This email and
> > its
> > contents are intended solely for the individual to whom it is addressed
> > and
> > may contain information that is confidential, privileged or otherwise
> > protected from disclosure, distributing or copying. Any views or
> opinions
> > presented in this email are solely those of the author and do not
> > necessarily represent those of Spicule Limited. The company accepts no
> > liability for any damage caused by any virus transmitted by this email.
> If
> > you have received this message in error, please notify us immediately by
> > reply email before deleting it from your system. Service of legal notice
> > cannot be effected on Spicule Limited by email.
> >
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
> This email and its contents are intended solely for the individual to whom
> it is addressed and may contain information that is confidential,
> privileged or otherwise protected from disclosure, distributing or copying.
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of Spicule Limited. The
> company accepts no liability for any damage caused by any virus transmitted
> by this email. If you have received this message in error, please notify us
> immediately by reply email before deleting it from your system. Service of
> legal notice cannot be effected on Spicule Limited by email.
>
Re: OODT docker builds
Posted by Tom Barber <to...@spicule.co.uk>.
I was thinking about the outputs. Currently everything is 1 docker file
which is fine for some deployments and not for others.
In the RADIX build we should also build individual containers for each
component. For the deployment we could also have a docker-compose file and
a K8S Helm setup so that you could deploy a distributed OODT setup from
your RADIX output, this could also have the ZK stuff in it so we can
properly utilise the distributed nature we started constructing with the FM
ZK changes.
On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha.13@cse.mrt.ac.lk)
wrote:
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> I’m interested in the Dockerization of OODT but also conscious that most
> people use RADIX to build their stuff, which make’s overriding bits of a
> prebuilt image tricky.
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
> add a Docker build step to the backend of people’s RADIX builds if they
so
> wanted.
>
> Thoughts?
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email.
If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Posted by Imesha Sudasingha <im...@cse.mrt.ac.lk>.
Hi Tom,
I think this will be great since people are adopting docker more and more
and even for a user
once they have built a customized docker image, they can share it among the
peers
reducing the time spent for configuration by each individual.
Also we have another option ;-) With distributed configuration management
which I implemented,
users can ask OODT components to download configuration published in
zookeeper. But this will
require zookeeper to be running (either as a container or standalone). As
per my understanding,
configuration is the problem we need to solve when using a pre-built docker
image?
In future, if we are able to implement
https://issues.apache.org/jira/browse/OODT-977
we will be able to run multiple file managers in multiple containers which
will allow
the load to be distributed and query all at once. So, docker will be the
way to go as I see it.
What do you think?
Thanks,
Imesha
On Sun, 14 Oct 2018 at 15:40, Tom Barber <to...@spicule.co.uk> wrote:
> I’m interested in the Dockerization of OODT but also conscious that most
> people use RADIX to build their stuff, which make’s overriding bits of a
> prebuilt image tricky.
>
> I’m wondering if its worth adding an optional Docker profile to RADIX to
> add a Docker build step to the backend of people’s RADIX builds if they so
> wanted.
>
> Thoughts?
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
> its
> contents are intended solely for the individual to whom it is addressed
> and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email. If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>