You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by 1ambda <gi...@git.apache.org> on 2016/12/14 14:40:42 UTC

[GitHub] zeppelin pull request #1761: [ZEPPELIN-1711] Create Released Binary Zeppelin...

GitHub user 1ambda opened a pull request:

    https://github.com/apache/zeppelin/pull/1761

    [ZEPPELIN-1711] Create Released Binary Zeppelin Docker Images

    ### What is this PR for?
    
    This PR
    
    - added Dockerfiles for released zeppelin binaries (0.6.0, 0.6.1, 0.6.2)
    - refactored Dockerfiles `scripts/docker/zeppelin-base` so that we can keep small size base images
    - added missing R, python packages to run tutorials properly
    - also included `scripts/docker/create-dockerfile.sh` to create docker images for newly released zeppelin binaries
    
    #### For Reviewers
    
    I have things to be discussed for this PR
    
    1. **Support other linux (centos) or not:** Alpine linux is designed for light weight OS so it doesn't have graphic devices. But usually people use zeppelin to render graph like things. In this case Alpine is not enough
    2. **Extracting script for installing required packages into `testing/install_external_dependencies.sh` or not:** Alpine linux has it's own command to install packages like `apk add`. I think this is not reusable. And extracting scripts from Dockerfile causes other problems. For example, we need to add script just for building base images for python, r to copy scripts for external dependencies because Docker command `COPY` and `ADD` doens't support relative paths. It means, we have to copy external dependency scripts before building into the path where Dockerfile is built.
    3. **Adding Binary Zeppelin Image to [docker-library/official-images](https://github.com/docker-library/official-images/)**: I haven't experiences about this.
    
    ### What type of PR is it?
    [Feature]
    
    ### What is the Jira issue?
    
    [ZEPPELIN-1711](https://issues.apache.org/jira/browse/ZEPPELIN-1711)
    
    ### How should this be tested?
    
    #### Building Base Images
    
    ```bash
    $ cd scripts/docker/zeppelin-base/alpine/java
    $ docker build . -t zeppelin:alpine-base_java
    
    $ cd scripts/docker/zeppelin-base/alpine/python
    $ docker build . -t zeppelin:alpine-base_python
    
    $ cd scripts/docker/zeppelin-base/alpine/r
    $ docker build . -t zeppelin:alpine-base_r
    ```
    
    #### Building Zeppelin Images
    
    ```bash
    $ scripts/docker/zeppelin-bin-all/alpine/0.6.2_java
    $ docker build . -t zeppelin:alpine-0.6.2_java
    
    $ scripts/docker/zeppelin-bin-all/alpine/0.6.2_python
    $ docker build . -t zeppelin:alpine-0.6.2_python
    
    $ scripts/docker/zeppelin-bin-all/alpine/0.6.2_r
    $ docker build . -t zeppelin:alpine-0.6.2_r
    
    # build 0.6.1, 0.6.0 images too
    ```
    
    #### Running Zeppelin Images
    
    Make sure you have zeppelin docker images before running containers
    
    ```bash
    $ docker images
    REPOSITORY               TAG                   IMAGE ID            CREATED             SIZE
    zeppelin                 alpine-base_java      5a24572968d6        5 hours ago         145 MB
    ```
    
    Then, running containers by replacing tags (0.6.2, 0.6.1, 0.6.0)
    
    ```
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_java
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_python
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_r
    
    # run 0.6.1, 0.6.0 also
    ```
    
    #### Testing Zeppelin Tutorials
    
    Here are things you need to know before testing.
    
    - `zeppelin:alpine-$TAG_java` images can run **Zeppelin Tutorial** since it has openjdk7
    - `zeppelin:alpine-$TAG_python` images can run **Zeppelin Tutorial: Python - matplotlib basic** as well as **Zeppelin Tutorial** since it has python related packages
    - `zeppelin:alpine-$TAG_r` images can run **R Tutorial** as well as **Zeppelin Tutorial** since it has python related packages
    - Since alpine linux doesn't have graphical device, some function may not work (e.g plot in R). Some R examples throw invalid images like chunked.png
    - For the same reason, we have to change some code of **Python - matplotlib basic** before testing otherwise it will throw tkinter errors in the python interpreter
    
    ```python
    // before
    %python
    import numpy as np
    import matplotlib.pyplot as plt
    
    // should be
    %python
    import numpy as np
    import matplotlib
    matplotlib.use('Agg')
    import matplotlib.pyplot as pet
    ```
    
    Then, run each zeppelin container and run tutorial while replacing tags by accessing localhost:8080 in your browsers
    
    ```
    # run: Zeppelin Tutorial
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_java
    
    # run: Zeppelin Tutorial, Python - matplotlib basic
    # and modify the code mentioned above before running paragraphs
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_python
    
    # run: Zeppelin Tutorial,  R Tutorial
    $ docker run -it --name zeppelin --rm -p 8080:8080 -p 7077:7077 zeppelin:alpine-0.6.2_r
    
    # test 0.6.1, 0.6.0 containers too
    ```
    
    #### Testing `create-dockerfile.sh`
    
    ```bash
    $ scripts/docker/zeppelin-bin-all/create-dockerfile.sh -h 
    
    USAGE: ./create-dockerfile.sh version linux platform
    * version: 0.6.2 (released zeppelin binary version)
    * linux: [alpine]
    * platform: [java, python, r]
    
    # For example,
    # This command will create `scripts/docker/zeppelin-bin-all/alpine/0.7.0_java/Dockerfile
    $ ./create-dockerfile.sh 0.7.0 alpine java
    ```
    
    
    ### Screenshots (if appropriate)
    
    #### Image Sizes
    
    zeppelin base image size added in #1538: 301.3 MB
    
    ```
    REPOSITORY               TAG                   IMAGE ID            CREATED             SIZE
    zeppelin                 alpine-base_java      5a24572968d6        5 hours ago         145 MB
    zeppelin                 alpine-base_python    e41dcf04760b        4 hours ago         684.9 MB
    zeppelin                 alpine-base_r         80c2d7aa7156        4 hours ago         538.9 MB
    zeppelin                 alpine-0.6.0_java     85b4502fd255        2 hours ago         737.7 MB
    zeppelin                 alpine-0.6.0_python   709dc585b94c        2 hours ago         1.278 GB
    zeppelin                 alpine-0.6.0_r        f6fba0c66563        2 hours ago         1.131 GB
    zeppelin                 alpine-0.6.1_java     ed95ca6204a9        3 hours ago         751.3 MB
    zeppelin                 alpine-0.6.1_python   34ceea2aeba8        3 hours ago         1.291 GB
    zeppelin                 alpine-0.6.1_r        3886596f3759        3 hours ago         1.145 GB
    zeppelin                 alpine-0.6.2_java     c65b2f6a1128        3 hours ago         786.4 MB
    zeppelin                 alpine-0.6.2_python   16c2c3630545        3 hours ago         1.326 GB
    zeppelin                 alpine-0.6.2_r        544311ec1e6a        3 hours ago         1.18 GB
    ```
    
    
    ### Questions:
    * Does the licenses files need update? - NO
    * Is there breaking changes for older versions? - NO
    * Does this needs documentation? - YES, I updated `docs/install/docker.md`
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/1ambda/zeppelin feat/docker-images

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/1761.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1761
    
----
commit 9f5968339cee5f8c3e03fa73cf5e5b1e10af9ff9
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T07:17:29Z

    feat: Install dumb-init using package

commit d9b8421499551500a7db9d23fdb504f84cb0c3d3
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T07:35:04Z

    feat: Install openjdk7
    
    ref - https://github.com/1ambda/docker-zeppelin/blob/master/alpine/base/Dockerfile

commit a9b3d67ec0df1a0b05627550e84f91b9549392ba
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T08:29:18Z

    chore: Use --no-cache not to remove cache dir

commit 085527d85c50533fccf089662fdf7c44fa3cccd2
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T08:33:05Z

    chore: Add logging

commit 24e15ef207dea4fe064978cbab826e131d2ce641
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T08:33:22Z

    fix: Remove exposed ports in base image

commit e8d2f5345f010bd5a866224825a351e0881ab01b
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T09:13:50Z

    feat: Add base_python, base_r

commit d7623f76e91182c0c570edf5515235ccce81f32a
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T09:53:49Z

    chore: Use zeppelin-base/alpine directory structure

commit 0730e72fe437248994bcef8e0a5b828d21adb38a
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T10:24:19Z

    feat: Add create-dockerfile.sh

commit ce7b2e42123bf05726dd2000ab37eafd03f73417
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T12:24:51Z

    fix: _java postfix

commit a10c304984b91af2fc820f9ea4a1917033b16d88
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:22:44Z

    fix: Base image directory structure

commit 03d32d93acd5bb5a03344904d1d7e15348171d27
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:23:04Z

    docs: Update docker docs

commit 0c297af9ff7bb83b10378b13427a4929ccdafbf3
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:35:44Z

    fix: Update create-dockerfile usage message

commit 5df6b7359ef0398a26cab478c1f1ab876fa9574d
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:36:13Z

    feat: Add 0.6.2 bin images

commit d960ffd2cba62ee9214100355a1d57502c146da9
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:36:40Z

    feat: Add 0.6.1 bin images

commit 4af7cebe25ec748ca1b04de086970888e6a42abb
Author: 1ambda <1a...@gmail.com>
Date:   2016-12-14T13:37:00Z

    feat: Add 0.6.0 bin images

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---