[jira] [Updated] (BIGTOP-2397) Add image pre-build new feature

["Evans Ye (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BIGTOP-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Evans Ye updated BIGTOP-2397:
-----------------------------
    Description: 
Currently the provisioner takes roughly 3 to 5 mins to get a cluster up and running. This is not quite efficient for CI and developers to get a cluster for use. This jira trys to add an image pre-build feature into docker provisioner so that users can build the image once and run it multiple times. With this we can expect a significant performance boost for provisioning.
BIGTOP-2296 showcased a solution which is to pre-build a hard-coded stack, i.e., hadoop + yarn. However, in this jira we'd like to make it more general, for example, user can define a stack named "in-memory stack", or "foobar company big data stack", which preload a set of components such as hadoop, yarn, spark into a image for further provisioning.

The following are more detailed steps for a cluster to be deployed with pre-build feature involved:

# user operation stage: update configuration file and run the provisioner command
{code}
vim config.yml 
# specify image name, for example *foobar_company:evans-hadoop-stack*, and the components
./docker-hadoop.sh --burn --create 3
# with --burn the image
{code}
# pre-build stage1: install general system packages such as java
# pre-build stage2: install stack packages such as hadoop, hbase, spark, etc
# provision stage1: upload configuration files
# provision stage2: run puppet apply to simply do deploy config files and start up services 

When user want to re-provision a clean cluster, the prebuild image will be used:
{code}
./docker-hadoop.sh --create 3 
{code}
There's actually no code change at the provisioner side, but with packages are already installed, the provision should be very fast.


  was:
Currently the provisioner takes roughly 3 to 5 mins to get a cluster up and running. This is not quite efficient for CI and developers to get a cluster for use. This jira trys to add an image pre-build feature into docker provisioner so that users can build the image once and run it multiple times. With this we can expect a substantial performance boost for provisioning.
BIGTOP-2296 showcased a solution which is to pre-build a hard-coded stack, i.e., hadoop + yarn. However, in this jira we'd like to make it more general, for example, user can define a stack named "in-memory stack", or "foobar company big data stack", which preload a set of components such as hadoop, yarn, spark into a image for further provisioning.

The following is the more detailed steps for a cluster to be deployed with pre-build feature involved:

# user operation stage: update configuration file and run the provisioner command
{code}
vim config.yml 
# specify image name, for example *foobar_company:evans-hadoop-stack*, and the components
./docker-hadoop.sh --burn --create 3
# with --burn the image
{code}
# pre-build stage1: install general system packages such as java
# pre-build stage2: install stack packages such as hadoop, hbase, spark, etc
# provision stage1: upload configuration files
# provision stage2: run puppet apply to simply do deploy config files and start up services 

When user want to re-provision a clean cluster, the prebuild image will be used:
{code}
./docker-hadoop.sh --create 3 
{code}
There's actually no code change at the provisioner side, but with packages are already installed, the provision should be very fast.



> Add image pre-build new feature
> -------------------------------
>
>                 Key: BIGTOP-2397
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-2397
>             Project: Bigtop
>          Issue Type: Sub-task
>          Components: docker, provisioner
>    Affects Versions: backlog
>            Reporter: Evans Ye
>            Assignee: Evans Ye
>
> Currently the provisioner takes roughly 3 to 5 mins to get a cluster up and running. This is not quite efficient for CI and developers to get a cluster for use. This jira trys to add an image pre-build feature into docker provisioner so that users can build the image once and run it multiple times. With this we can expect a significant performance boost for provisioning.
> BIGTOP-2296 showcased a solution which is to pre-build a hard-coded stack, i.e., hadoop + yarn. However, in this jira we'd like to make it more general, for example, user can define a stack named "in-memory stack", or "foobar company big data stack", which preload a set of components such as hadoop, yarn, spark into a image for further provisioning.
> The following are more detailed steps for a cluster to be deployed with pre-build feature involved:
> # user operation stage: update configuration file and run the provisioner command
> {code}
> vim config.yml 
> # specify image name, for example *foobar_company:evans-hadoop-stack*, and the components
> ./docker-hadoop.sh --burn --create 3
> # with --burn the image
> {code}
> # pre-build stage1: install general system packages such as java
> # pre-build stage2: install stack packages such as hadoop, hbase, spark, etc
> # provision stage1: upload configuration files
> # provision stage2: run puppet apply to simply do deploy config files and start up services 
> When user want to re-provision a clean cluster, the prebuild image will be used:
> {code}
> ./docker-hadoop.sh --create 3 
> {code}
> There's actually no code change at the provisioner side, but with packages are already installed, the provision should be very fast.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)