[jira] [Created] (BIGTOP-2397) Add image pre-build new feature

["Evans Ye (JIRA)" <ji...@apache.org>]

Evans Ye created BIGTOP-2397:
--------------------------------

             Summary: Add image pre-build new feature
                 Key: BIGTOP-2397
                 URL: https://issues.apache.org/jira/browse/BIGTOP-2397
             Project: Bigtop
          Issue Type: Sub-task
          Components: docker, provisioner
    Affects Versions: backlog
            Reporter: Evans Ye
            Assignee: Evans Ye


Currently the provisioner takes roughly 3 to 5 mins to get a cluster up and running. This is not quite efficient for CI and developers to get a cluster for use. This jira trys to add an image pre-build feature into docker provisioner so that users can build the image once and run it multiple times. With this we can expect a substantial performance boost for provisioning.
BIGTOP-2296 showcased a solution which is to pre-build a hard-coded stack, i.e., hadoop + yarn. However, in this jira we'd like to make it more general, for example, user can define a stack named "in-memory stack", or "foobar company big data stack", which preload a set of components such as hadoop, yarn, spark into a image for further provisioning.

The following is the more detailed steps for a cluster to be deployed with pre-build feature involved:

# user operation stage: update configuration file and run the provisioner command
{code}
vim config.yml 
# specify image name, for example *foobar_company:evans-hadoop-stack*, and the components
./docker-hadoop.sh --burn --create 3
# with --burn the image
{code}
# pre-build stage1: install general system packages such as java
# pre-build stage2: install stack packages such as hadoop, hbase, spark, etc
# provision stage1: upload configuration files
# provision stage2: run puppet apply to simply do deploy config files and start up services 

When user want to re-provision a clean cluster, the prebuild image will be used:
{code}
./docker-hadoop.sh --create 3 
{code}
There's actually no code change at the provisioner side, but with packages are already installed, the provision should be very fast.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)