Re: [Discuss] OODT Containerization Plan

[Madhawa Vidanapathirana <ma...@gmail.com>]

Hi all,

Looks like a very interesting plan.
I have been working with Docker and docker-compose for about a year now and
I would like to contribute. I am fairly new to Kubernetes though but still,
I might be able to help on that part as well.

Kind regards,
Madhawa

*Madhawa Vidanapathirana*
Student
Department of Computer Science and Engineering
University of Moratuwa
Sri Lanka

Mobile: (+94) 716874425
Email: madhawavidanapathirana@gmail.com
Linked-In: *https://www.linkedin.com/in/madhawav/
<https://www.linkedin.com/in/madhawav/>*


On Fri, Nov 30, 2018 at 4:00 PM Imesha Sudasingha <im...@cse.mrt.ac.lk>
wrote:

> Hi Lewis,
>
> The documentation related to distributed configuration management is at:
>
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> .
> But it is not available in 1.2.x releases.
>
> This tool lets you upload configuration (say policies, properties files,
> etc) to filemanagers, resource managers and workflow managers without
> knowing their IPs. If you have a multi node cluster and want to publish
> common configuration to all of them, this tool can be used. The difference
> compared to something like ansible and this is, you don't need to know the
> IPs of the nodes.
>
> Furthermore, I have tested the ability to listen for configuration change
> and apply them accordingly to file manager. Try them out and let me know
> what you think or any improvement is required.
>
> Thanks,
> Imesha
>
>
>
>
> On Fri, 30 Nov 2018 at 01:38, Tom Barber <to...@spicule.co.uk> wrote:
>
> > Imesha is the king of the distributed config stuff, I believe some docs
> > went in the wiki although I can’t check right now.
> >
> > Interesting stuff on Brooklyn somehow I’d missed that, gonna check it
> out!
> >
> >
> > On 29 November 2018 at 19:18:06, Lewis John McGibbney (
> lewismc@apache.org)
> > wrote:
> >
> > Some recent work on my end on a similar thread, for COAL-SDS [0] which is
> > essentially a customized RADiX deployment, right now I'm working on
> writing
> > Apache Brooklyn [1] blueprints which will really help me to deploy
> COAL-SDS
> > to any cloud platform with the necessary dependencies installed so I can
> > execute the algorithms which run in my OODT workflow.
> > I didn't want to tie this to any particular cloud provider hence the
> > Brooklyn blueprints.
> >
> > Back to the topic of conversation here, I think we should start with the
> > basic services e.g. file, workflow and resource mgr. These can be
> > Dockerized very easily with necessary flags for service configuration.
> > Admitedly I
> > ve not tried out the distributed configuration management yet. Is there
> any
> > documentation I can swot up on it please?
> >
> > [0] https://github.com/capstone-coal/coal-sds/
> > [1] https://brooklyn.apache.org
> >
> > On 2018/11/24 22:22:06, Tom Barber <to...@spicule.co.uk> wrote:
> > > Hey Imesha, et al,
> > >
> > > I think the ideas are pretty good. Radix is a good starting point, but
> > > there needs to be a useful recipe for people wanting to customise it
> with
> > > their own stuff.
> > >
> > > Docker Compose is a good way for spinning up a multi container
> solution,
> > > although without Swarm, you don’t get any redundancy. Kubernetes will
> > give
> > > users better scale out support. I’m also testing some cool Juju support
> > for
> > > Kubernetes which might make the deployment of this stuff easier on K8S.
> > >
> > > I’d certainly like to see an out of the box solution which uses the ZK
> > > stuff you did, the Avro stuff thats going in and OODT, even if its a
> way
> > > for us to discover the ZK and scale up limitations. I think using
> Docker
> > > will make larger deployments of OODT easier and something we should
> > > certainly put some thought into.
> > >
> > >
> > > On 18 November 2018 at 11:53:53, Imesha Sudasingha (
> > imesha.13@cse.mrt.ac.lk)
> > > wrote:
> > >
> > > Hi All,
> > >
> > > Recently we had an interesting discussion
> > > <
> > >
> >
> >
> https://lists.apache.org/thread.html/759f2145d9eacf87f3ed1255236354d713633883b47b91249b1f5c75@
> > > <dev.oodt.apache.org>>
> > > [1] on making OODT easy to use out of the box through containerization.
> > As
> > > a summary I suggest following steps to move on.
> > >
> > > 1. Implement ability to build component wise docker images. Tom's idea
> > was
> > > to add a docker build profile to RADIX build. I think this will be a
> good
> > > starting point.
> > >
> > > 2. Add easy deployment capability through docker-compose
> > > <https://docs.docker.com/compose/> and kubernetes. We can keep
> > kubernetes
> > > for later depending on the effort we can put. This will provide any new
> > > comer to run a working example of OODT with a single command. A big
> plus
> > > point from new comers' point of view.
> > >
> > > 3. Since we have distributed configuration management
> > > <
> > >
> >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> > >
> >
> > >
> > > [2] already implemented, may be we don't need to put any burden of
> > building
> > > docker images to users. Instead, they can simply start a generic OODT
> > > deployment (say using docker compose) and publish custom configuration
> > > (properties, policies, etc) to those running components. However, need
> to
> > > discuss further on this.
> > >
> > > What do you think? Comments?
> > >
> > > New contributors are also welcome to participate in this discussion
> (and
> > of
> > > course to contribute ;-) )
> > >
> > > Cheers,
> > > Imesha
> > >
> > > [1]
> > >
> >
> >
> https://lists.apache.org/thread.html/759f2145d9eacf87f3ed1255236354d713633883b47b91249b1f5c75@%3Cdev.oodt.apache.org%3E
> > > [2]
> > >
> >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> > >
> > > --
> > >
> > >
> > > Spicule Limited is registered in England & Wales. Company Number:
> > > 09954122. Registered office: First Floor, Telecom House, 125-135
> Preston
> > > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> > >
> > >
> > >
> > >
> > > All engagements
> > > are subject to Spicule Terms and Conditions of Business. This email and
> > its
> > > contents are intended solely for the individual to whom it is addressed
> > and
> > > may contain information that is confidential, privileged or otherwise
> > > protected from disclosure, distributing or copying. Any views or
> opinions
> > > presented in this email are solely those of the author and do not
> > > necessarily represent those of Spicule Limited. The company accepts no
> > > liability for any damage caused by any virus transmitted by this email.
> > If
> > > you have received this message in error, please notify us immediately
> by
> > > reply email before deleting it from your system. Service of legal
> notice
> > > cannot be effected on Spicule Limited by email.
> > >
> >
> > --
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> > All engagements
> > are subject to Spicule Terms and Conditions of Business. This email and
> > its
> > contents are intended solely for the individual to whom it is addressed
> > and
> > may contain information that is confidential, privileged or otherwise
> > protected from disclosure, distributing or copying. Any views or opinions
> > presented in this email are solely those of the author and do not
> > necessarily represent those of Spicule Limited. The company accepts no
> > liability for any damage caused by any virus transmitted by this email.
> If
> > you have received this message in error, please notify us immediately by
> > reply email before deleting it from your system. Service of legal notice
> > cannot be effected on Spicule Limited by email.
> >
>

Re: [Discuss] OODT Containerization Plan

[Imesha Sudasingha <im...@cse.mrt.ac.lk>]

Hi Madhawa,

Thanks for your interest. We could really use your help for this.
As a starting point, I thought of coming up with images per component and
add that to the maven build.
Can you help with that?

Thanks,
Imesha


On Thu, 6 Dec 2018 at 21:48, Madhawa Vidanapathirana <
madhawavidanapathirana@gmail.com> wrote:

> Hi all,
>
> Looks like a very interesting plan.
> I have been working with Docker and docker-compose for about a year now and
> I would like to contribute. I am fairly new to Kubernetes though but still,
> I might be able to help on that part as well.
>
> Kind regards,
> Madhawa
>
> *Madhawa Vidanapathirana*
> Student
> Department of Computer Science and Engineering
> University of Moratuwa
> Sri Lanka
>
> Mobile: (+94) 716874425
> Email: madhawavidanapathirana@gmail.com
> Linked-In: *https://www.linkedin.com/in/madhawav/
> <https://www.linkedin.com/in/madhawav/>*
>
>
> On Fri, Nov 30, 2018 at 4:00 PM Imesha Sudasingha <imesha.13@cse.mrt.ac.lk
> >
> wrote:
>
> > Hi Lewis,
> >
> > The documentation related to distributed configuration management is at:
> >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> > .
> > But it is not available in 1.2.x releases.
> >
> > This tool lets you upload configuration (say policies, properties files,
> > etc) to filemanagers, resource managers and workflow managers without
> > knowing their IPs. If you have a multi node cluster and want to publish
> > common configuration to all of them, this tool can be used. The
> difference
> > compared to something like ansible and this is, you don't need to know
> the
> > IPs of the nodes.
> >
> > Furthermore, I have tested the ability to listen for configuration change
> > and apply them accordingly to file manager. Try them out and let me know
> > what you think or any improvement is required.
> >
> > Thanks,
> > Imesha
> >
> >
> >
> >
> > On Fri, 30 Nov 2018 at 01:38, Tom Barber <to...@spicule.co.uk> wrote:
> >
> > > Imesha is the king of the distributed config stuff, I believe some docs
> > > went in the wiki although I can’t check right now.
> > >
> > > Interesting stuff on Brooklyn somehow I’d missed that, gonna check it
> > out!
> > >
> > >
> > > On 29 November 2018 at 19:18:06, Lewis John McGibbney (
> > lewismc@apache.org)
> > > wrote:
> > >
> > > Some recent work on my end on a similar thread, for COAL-SDS [0] which
> is
> > > essentially a customized RADiX deployment, right now I'm working on
> > writing
> > > Apache Brooklyn [1] blueprints which will really help me to deploy
> > COAL-SDS
> > > to any cloud platform with the necessary dependencies installed so I
> can
> > > execute the algorithms which run in my OODT workflow.
> > > I didn't want to tie this to any particular cloud provider hence the
> > > Brooklyn blueprints.
> > >
> > > Back to the topic of conversation here, I think we should start with
> the
> > > basic services e.g. file, workflow and resource mgr. These can be
> > > Dockerized very easily with necessary flags for service configuration.
> > > Admitedly I
> > > ve not tried out the distributed configuration management yet. Is there
> > any
> > > documentation I can swot up on it please?
> > >
> > > [0] https://github.com/capstone-coal/coal-sds/
> > > [1] https://brooklyn.apache.org
> > >
> > > On 2018/11/24 22:22:06, Tom Barber <to...@spicule.co.uk> wrote:
> > > > Hey Imesha, et al,
> > > >
> > > > I think the ideas are pretty good. Radix is a good starting point,
> but
> > > > there needs to be a useful recipe for people wanting to customise it
> > with
> > > > their own stuff.
> > > >
> > > > Docker Compose is a good way for spinning up a multi container
> > solution,
> > > > although without Swarm, you don’t get any redundancy. Kubernetes will
> > > give
> > > > users better scale out support. I’m also testing some cool Juju
> support
> > > for
> > > > Kubernetes which might make the deployment of this stuff easier on
> K8S.
> > > >
> > > > I’d certainly like to see an out of the box solution which uses the
> ZK
> > > > stuff you did, the Avro stuff thats going in and OODT, even if its a
> > way
> > > > for us to discover the ZK and scale up limitations. I think using
> > Docker
> > > > will make larger deployments of OODT easier and something we should
> > > > certainly put some thought into.
> > > >
> > > >
> > > > On 18 November 2018 at 11:53:53, Imesha Sudasingha (
> > > imesha.13@cse.mrt.ac.lk)
> > > > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Recently we had an interesting discussion
> > > > <
> > > >
> > >
> > >
> >
> https://lists.apache.org/thread.html/759f2145d9eacf87f3ed1255236354d713633883b47b91249b1f5c75@
> > > > <dev.oodt.apache.org>>
> > > > [1] on making OODT easy to use out of the box through
> containerization.
> > > As
> > > > a summary I suggest following steps to move on.
> > > >
> > > > 1. Implement ability to build component wise docker images. Tom's
> idea
> > > was
> > > > to add a docker build profile to RADIX build. I think this will be a
> > good
> > > > starting point.
> > > >
> > > > 2. Add easy deployment capability through docker-compose
> > > > <https://docs.docker.com/compose/> and kubernetes. We can keep
> > > kubernetes
> > > > for later depending on the effort we can put. This will provide any
> new
> > > > comer to run a working example of OODT with a single command. A big
> > plus
> > > > point from new comers' point of view.
> > > >
> > > > 3. Since we have distributed configuration management
> > > > <
> > > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> > > >
> > >
> > > >
> > > > [2] already implemented, may be we don't need to put any burden of
> > > building
> > > > docker images to users. Instead, they can simply start a generic OODT
> > > > deployment (say using docker compose) and publish custom
> configuration
> > > > (properties, policies, etc) to those running components. However,
> need
> > to
> > > > discuss further on this.
> > > >
> > > > What do you think? Comments?
> > > >
> > > > New contributors are also welcome to participate in this discussion
> > (and
> > > of
> > > > course to contribute ;-) )
> > > >
> > > > Cheers,
> > > > Imesha
> > > >
> > > > [1]
> > > >
> > >
> > >
> >
> https://lists.apache.org/thread.html/759f2145d9eacf87f3ed1255236354d713633883b47b91249b1f5c75@%3Cdev.oodt.apache.org%3E
> > > > [2]
> > > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management
> > > >
> > > > --
> > > >
> > > >
> > > > Spicule Limited is registered in England & Wales. Company Number:
> > > > 09954122. Registered office: First Floor, Telecom House, 125-135
> > Preston
> > > > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> > > >
> > > >
> > > >
> > > >
> > > > All engagements
> > > > are subject to Spicule Terms and Conditions of Business. This email
> and
> > > its
> > > > contents are intended solely for the individual to whom it is
> addressed
> > > and
> > > > may contain information that is confidential, privileged or otherwise
> > > > protected from disclosure, distributing or copying. Any views or
> > opinions
> > > > presented in this email are solely those of the author and do not
> > > > necessarily represent those of Spicule Limited. The company accepts
> no
> > > > liability for any damage caused by any virus transmitted by this
> email.
> > > If
> > > > you have received this message in error, please notify us immediately
> > by
> > > > reply email before deleting it from your system. Service of legal
> > notice
> > > > cannot be effected on Spicule Limited by email.
> > > >
> > >
> > > --
> > >
> > >
> > > Spicule Limited is registered in England & Wales. Company Number:
> > > 09954122. Registered office: First Floor, Telecom House, 125-135
> Preston
> > > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> > >
> > >
> > >
> > >
> > > All engagements
> > > are subject to Spicule Terms and Conditions of Business. This email and
> > > its
> > > contents are intended solely for the individual to whom it is addressed
> > > and
> > > may contain information that is confidential, privileged or otherwise
> > > protected from disclosure, distributing or copying. Any views or
> opinions
> > > presented in this email are solely those of the author and do not
> > > necessarily represent those of Spicule Limited. The company accepts no
> > > liability for any damage caused by any virus transmitted by this email.
> > If
> > > you have received this message in error, please notify us immediately
> by
> > > reply email before deleting it from your system. Service of legal
> notice
> > > cannot be effected on Spicule Limited by email.
> > >
> >
>