[GitHub] storm pull request #2901: [STORM-3271] Docker support: launch storm workers ...

[Ethanlm <gi...@git.apache.org>]

GitHub user Ethanlm opened a pull request:

    https://github.com/apache/storm/pull/2901

    [STORM-3271] Docker support:  launch storm workers in docker containers

    Spent a lot of effort on this.
    
    This patch let storm supervisor launches the worker in a docker container so that user code is isolated from each other and from the host machine. This is a security enhancement.
    
    Detailed explanation of this are available at docs/Docker-support.md in this pr.
    
    I tested it manually (only on `RHEL7`, didn't test it on other os systems) and tested with secure cluster setup. Also tested with our own integration test suites with docker support turned off and proved that this pr won't break storm.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Ethanlm/storm STORM-3271

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/2901.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2901
    
----
commit ba8b7cf80347fe077e097206023adb3326c8b4ad
Author: Ethan Li <et...@...>
Date:   2018-10-12T23:24:15Z

    refactor supervisor source code

commit 2387da9ae554ed81573e35545992754bc1454c9d
Author: Ethan Li <et...@...>
Date:   2018-10-12T23:24:23Z

    add docker support

commit b130bcb649f6adcb1fef588dc7033361999a9d0f
Author: Ethan Li <et...@...>
Date:   2018-10-15T15:27:39Z

    add cgrouproot and nscd bind mounts
    
    add user's groups and simply getGroupIdInfo and getUserIdInfo functions

commit be2efc833144004c126ff150f9676adc15575549
Author: Ethan Li <et...@...>
Date:   2018-10-16T13:48:16Z

    address some review comments

commit 80f3296688fce2bec9692a2699ae396a32c8e02e
Author: Ethan Li <et...@...>
Date:   2018-10-16T22:04:15Z

    fix worker-launcher for secure storm cluster

commit d4e2127ef9664c0398f2e85c26d0dde5734d3444
Author: Ethan Li <et...@...>
Date:   2018-10-17T14:35:19Z

    launch docker container in attached mode

commit 8e4c3c7a3cfe2a38dc847ea2ebda2ce74f9a45ff
Author: Ethan Li <et...@...>
Date:   2018-10-17T16:05:18Z

    check prerequisites in prepare method; bind mount workerArtifact more precisely

commit e0131136fd864aa5bc4a3b8e40b538d04a1c36ae
Author: Ethan Li <et...@...>
Date:   2018-10-19T19:57:24Z

    fix user could be null (e.g when supervisor restarts)

commit 5dfcd9ff8869f6a4334ccecdefb0f5c1f40ea9fb
Author: Ethan Li <et...@...>
Date:   2018-10-22T20:24:20Z

    fix typo; make network type non-configurable

commit fa5e2999312258b4da0b1936591804780bd8ec8e
Author: Ethan Li <et...@...>
Date:   2018-10-23T19:55:45Z

    make worker-launcher more secure

commit 0f08bcde454f7a2e613eb673ba697236d8122999
Author: Ethan Li <et...@...>
Date:   2018-10-24T16:28:24Z

    make nsenter more secure

commit 721c6b85e38b87761f0a52c19cd9b684bf70cbba
Author: Ethan Li <et...@...>
Date:   2018-10-24T20:10:56Z

    add more comments; add list of readonly bindmounts

commit be88fb7c174ab15e22ab99590530919499fe9d38
Author: Ethan Li <et...@...>
Date:   2018-10-24T20:53:10Z

    making nsenter safer

commit 1d84bf7d86cfa7e3c603ef4f99e0e6c4e2ec9ede
Author: Ethan Li <et...@...>
Date:   2018-10-25T14:23:18Z

    add a whitelist for docker images

commit 9c8ae38bdbc31d897eca9f1779d96d5b995fcec6
Author: Ethan Li <et...@...>
Date:   2018-10-25T21:20:39Z

    check every image in the whitelist; prevent memory leark from worker-launcher; fix format and trivial issues.

commit 52350f4363821ac14a5b7b2114bc1bddb204299a
Author: Ethan Li <et...@...>
Date:   2018-10-26T20:18:25Z

    launch docker container in background; use docker-wait to get the exitCode if the contaner exits

commit 6c80da77481cccabd047452b055ab3cd68a67c40
Author: Ethan Li <et...@...>
Date:   2018-10-30T20:46:26Z

    add a cgroup sub path template to support various cgroup setup

commit 8b9c18f314c1f8bf12140631f7de052bfc35cff5
Author: Ethan Li <et...@...>
Date:   2018-10-31T20:19:13Z

    add unit tests for docker commands

commit 05b31e875536f01b3fb61a378ce9de3c01d4a5d3
Author: Ethan Li <et...@...>
Date:   2018-11-01T13:51:00Z

    use nsenter to do profiling; mount shared_by_topology/tmp to /tmp

commit a1ffe50265f446b68dac0350c8cb4c49bf05992b
Author: Ethan Li <et...@...>
Date:   2018-11-01T15:02:12Z

    fix loging for storm.resource.isolation.plugin

commit c4647183092541dc5957441500f62719510689f5
Author: Ethan Li <et...@...>
Date:   2018-11-01T17:38:54Z

    fix issues when DockerManager is not used

commit 190f06938a38f4225d94f86c2f1afbba068d1b22
Author: Ethan Li <et...@...>
Date:   2018-11-05T16:54:32Z

    add documentation

----


---