You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by na...@reactor8.com on 2016/04/02 02:31:57 UTC
adding new deployment metadata files to bigtop
Hello bigtoppers,
Our org has been leveraging bigtop with our consulting work the past couple
years, primarily focused on deploying/managing clustered services using the
bigtop puppet modules. We typically will pull periodic updates and merge in
latest bigtop puppet changes with our version. Recently we have had couple
customer and community inquires interested in using bigtop to start to play
with various builds and component versions, having us integrate our
deployment stuff. We are initially going to maintain fork of bigtop to get
the ball rolling on a couple projects, but are interested in adding some
base set of our meta files to bigtop so people don't have to come to us for
our fork, instead just maintain and use vanilla apache bigtop.
The files we would like to contribute and maintain could be considered akin
to a vagrant or docker compose file, but instead of describing single
node/image, it describes a distributed service/cluster that can run any
number of services, leveraging the puppet modules for configuration.
Here is partial reference description for a spark service:
https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
We are going to start prototyping in the next week or two, looking at a file
or two in the root folder, then probably unique folder in the root that can
house various service meta descriptions so people can just use off the shelf
without having to know details or write their own by default. In general
the files are just text/yaml files, and will have no disruption to existing
gradle/tooling.
In meantime until we have something tangible, wanted to start the
conversation for community to provide any questions/feedback/etc about
approach of including a base set of service meta files
Nate
RE: adding new deployment metadata files to bigtop
Posted by na...@reactor8.com.
Some answers inline:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
Correct, meta file is for external usage, like a vagrant file which sits in bigtop dir, but is only used externally by vagrant to deploy a node. Difference is this instead of defining a single node/vm/docker-image, etc, it’s typically defining a distributed topology/cluster, and the workflow system would handle node provisioning and installing the bigtop packages. In regards to puppet question, right now there are two metadata descriptions, one that describes topologies called an assembly, the other describes "components" which acts as a interface def in bigtops case to the puppet modules so the system can load and execute the modules to handle the config and service start part of it.
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
I think in initial implementation the benefit would mostly be for people deploying distributed clusters in environments like aws as the workflow system can handle automating the node deployments, networking, and software install/config. Instead of having just a single monolithic docker image, or equivalent, can have something like "spark+hdfs+zeppelin cluster" with appropriate master/worker design. Right now the workflow system is bundled as starter uber docker all-in-one container, kind of like the bigtop build image.
For local laptop/dev usage, will take little thinking/work, but we can probably look at making a "localhost docker provider" that would be analogous to aws provider that could spin up and execute the workflow all on localhost using docker for node deployment vs aws.
-----Original Message-----
From: Konstantin Boudnik [mailto:cos@apache.org]
Sent: Wednesday, April 6, 2016 7:57 AM
To: user@bigtop.apache.org; dev@bigtop.apache.org
Subject: Re: adding new deployment metadata files to bigtop
Hi Nate.
Thanks for the decision to share the work you've been doing and help this community to advance. I want to clarify a couple of questions here:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
In general, I am very fond of having an orchestration in Bigtop, done one way or another.
Thanks!
Cos
On Fri, Apr 01, 2016 at 05:31PM, nate@reactor8.com wrote:
> Hello bigtoppers,
>
> Our org has been leveraging bigtop with our consulting work the past
> couple years, primarily focused on deploying/managing clustered
> services using the bigtop puppet modules. We typically will pull
> periodic updates and merge in latest bigtop puppet changes with our
> version. Recently we have had couple customer and community inquires
> interested in using bigtop to start to play with various builds and
> component versions, having us integrate our deployment stuff. We are
> initially going to maintain fork of bigtop to get the ball rolling on
> a couple projects, but are interested in adding some base set of our
> meta files to bigtop so people don't have to come to us for our fork, instead just maintain and use vanilla apache bigtop.
>
> The files we would like to contribute and maintain could be considered
> akin to a vagrant or docker compose file, but instead of describing
> single node/image, it describes a distributed service/cluster that can
> run any number of services, leveraging the puppet modules for configuration.
>
> Here is partial reference description for a spark service:
>
> https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
>
> We are going to start prototyping in the next week or two, looking at
> a file or two in the root folder, then probably unique folder in the
> root that can house various service meta descriptions so people can
> just use off the shelf without having to know details or write their
> own by default. In general the files are just text/yaml files, and
> will have no disruption to existing gradle/tooling.
>
> In meantime until we have something tangible, wanted to start the
> conversation for community to provide any questions/feedback/etc about
> approach of including a base set of service meta files
RE: adding new deployment metadata files to bigtop
Posted by na...@reactor8.com.
Some answers inline:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
Correct, meta file is for external usage, like a vagrant file which sits in bigtop dir, but is only used externally by vagrant to deploy a node. Difference is this instead of defining a single node/vm/docker-image, etc, it’s typically defining a distributed topology/cluster, and the workflow system would handle node provisioning and installing the bigtop packages. In regards to puppet question, right now there are two metadata descriptions, one that describes topologies called an assembly, the other describes "components" which acts as a interface def in bigtops case to the puppet modules so the system can load and execute the modules to handle the config and service start part of it.
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
I think in initial implementation the benefit would mostly be for people deploying distributed clusters in environments like aws as the workflow system can handle automating the node deployments, networking, and software install/config. Instead of having just a single monolithic docker image, or equivalent, can have something like "spark+hdfs+zeppelin cluster" with appropriate master/worker design. Right now the workflow system is bundled as starter uber docker all-in-one container, kind of like the bigtop build image.
For local laptop/dev usage, will take little thinking/work, but we can probably look at making a "localhost docker provider" that would be analogous to aws provider that could spin up and execute the workflow all on localhost using docker for node deployment vs aws.
-----Original Message-----
From: Konstantin Boudnik [mailto:cos@apache.org]
Sent: Wednesday, April 6, 2016 7:57 AM
To: user@bigtop.apache.org; dev@bigtop.apache.org
Subject: Re: adding new deployment metadata files to bigtop
Hi Nate.
Thanks for the decision to share the work you've been doing and help this community to advance. I want to clarify a couple of questions here:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
In general, I am very fond of having an orchestration in Bigtop, done one way or another.
Thanks!
Cos
On Fri, Apr 01, 2016 at 05:31PM, nate@reactor8.com wrote:
> Hello bigtoppers,
>
> Our org has been leveraging bigtop with our consulting work the past
> couple years, primarily focused on deploying/managing clustered
> services using the bigtop puppet modules. We typically will pull
> periodic updates and merge in latest bigtop puppet changes with our
> version. Recently we have had couple customer and community inquires
> interested in using bigtop to start to play with various builds and
> component versions, having us integrate our deployment stuff. We are
> initially going to maintain fork of bigtop to get the ball rolling on
> a couple projects, but are interested in adding some base set of our
> meta files to bigtop so people don't have to come to us for our fork, instead just maintain and use vanilla apache bigtop.
>
> The files we would like to contribute and maintain could be considered
> akin to a vagrant or docker compose file, but instead of describing
> single node/image, it describes a distributed service/cluster that can
> run any number of services, leveraging the puppet modules for configuration.
>
> Here is partial reference description for a spark service:
>
> https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
>
> We are going to start prototyping in the next week or two, looking at
> a file or two in the root folder, then probably unique folder in the
> root that can house various service meta descriptions so people can
> just use off the shelf without having to know details or write their
> own by default. In general the files are just text/yaml files, and
> will have no disruption to existing gradle/tooling.
>
> In meantime until we have something tangible, wanted to start the
> conversation for community to provide any questions/feedback/etc about
> approach of including a base set of service meta files
Re: adding new deployment metadata files to bigtop
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Nate.
Thanks for the decision to share the work you've been doing and help this
community to advance. I want to clarify a couple of questions here:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
In general, I am very fond of having an orchestration in Bigtop, done one way
or another.
Thanks!
Cos
On Fri, Apr 01, 2016 at 05:31PM, nate@reactor8.com wrote:
> Hello bigtoppers,
>
> Our org has been leveraging bigtop with our consulting work the past couple
> years, primarily focused on deploying/managing clustered services using the
> bigtop puppet modules. We typically will pull periodic updates and merge in
> latest bigtop puppet changes with our version. Recently we have had couple
> customer and community inquires interested in using bigtop to start to play
> with various builds and component versions, having us integrate our
> deployment stuff. We are initially going to maintain fork of bigtop to get
> the ball rolling on a couple projects, but are interested in adding some
> base set of our meta files to bigtop so people don't have to come to us for
> our fork, instead just maintain and use vanilla apache bigtop.
>
> The files we would like to contribute and maintain could be considered akin
> to a vagrant or docker compose file, but instead of describing single
> node/image, it describes a distributed service/cluster that can run any
> number of services, leveraging the puppet modules for configuration.
>
> Here is partial reference description for a spark service:
>
> https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
>
> We are going to start prototyping in the next week or two, looking at a file
> or two in the root folder, then probably unique folder in the root that can
> house various service meta descriptions so people can just use off the shelf
> without having to know details or write their own by default. In general
> the files are just text/yaml files, and will have no disruption to existing
> gradle/tooling.
>
> In meantime until we have something tangible, wanted to start the
> conversation for community to provide any questions/feedback/etc about
> approach of including a base set of service meta files
Re: adding new deployment metadata files to bigtop
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Nate.
Thanks for the decision to share the work you've been doing and help this
community to advance. I want to clarify a couple of questions here:
- the metadata files are presumably getting executed by some sort of external
engine doing actual orchestration work, right? These files are, please
forgive me for the lack of the better word or analogy, carry the
function of topology map, and the actual installation/configuration of the
nodes is still done by the good ol' Puppet. Is this correct?
- with these files in place, how one can benefit from them? Is there a service
that allow to plug them in? Or an orchestration engine, that could be ran on
my laptop (or else)? Basically, I am trying to figure out what other pieces
will be there and how they all get together.
In general, I am very fond of having an orchestration in Bigtop, done one way
or another.
Thanks!
Cos
On Fri, Apr 01, 2016 at 05:31PM, nate@reactor8.com wrote:
> Hello bigtoppers,
>
> Our org has been leveraging bigtop with our consulting work the past couple
> years, primarily focused on deploying/managing clustered services using the
> bigtop puppet modules. We typically will pull periodic updates and merge in
> latest bigtop puppet changes with our version. Recently we have had couple
> customer and community inquires interested in using bigtop to start to play
> with various builds and component versions, having us integrate our
> deployment stuff. We are initially going to maintain fork of bigtop to get
> the ball rolling on a couple projects, but are interested in adding some
> base set of our meta files to bigtop so people don't have to come to us for
> our fork, instead just maintain and use vanilla apache bigtop.
>
> The files we would like to contribute and maintain could be considered akin
> to a vagrant or docker compose file, but instead of describing single
> node/image, it describes a distributed service/cluster that can run any
> number of services, leveraging the puppet modules for configuration.
>
> Here is partial reference description for a spark service:
>
> https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712
>
> We are going to start prototyping in the next week or two, looking at a file
> or two in the root folder, then probably unique folder in the root that can
> house various service meta descriptions so people can just use off the shelf
> without having to know details or write their own by default. In general
> the files are just text/yaml files, and will have no disruption to existing
> gradle/tooling.
>
> In meantime until we have something tangible, wanted to start the
> conversation for community to provide any questions/feedback/etc about
> approach of including a base set of service meta files