You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/06/12 17:02:52 UTC
[GitHub] [storm] Ethanlm opened a new pull request #3287: [STORM-3271] enable supervisors to launch a worker inside a docker container
Ethanlm opened a new pull request #3287:
URL: https://github.com/apache/storm/pull/3287
## What is the purpose of the change
With this new functionality, a supervisor can launch a worker process inside a docker container for better security and portability.
## How was the change tested
1. Tested by submitting a topology, and validated the basic functions like launching, killing, cleaning up, profiling works
2. Validated the worker process is isolated inside a container with mostly read-only filesystem, and its own pid namespace
3. Validated cpu bindings and cpu throttling work as expected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3287: [STORM-3271] enable supervisors to launch a worker inside a docker container
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3287:
URL: https://github.com/apache/storm/pull/3287#discussion_r450424051
##########
File path: storm-server/src/main/java/org/apache/storm/container/docker/DockerPsCommand.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.container.docker;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang.StringUtils;
+
+public class DockerPsCommand extends DockerCommand {
+ private static final String PS_COMMAND = "ps";
+
+ public DockerPsCommand() {
+ super(PS_COMMAND);
+ }
+
+ public DockerPsCommand withQuietOption() {
+ super.addCommandArguments("--quiet=true");
+ return this;
+ }
+
+ public DockerPsCommand withNameFilter(String containerName) {
+ super.addCommandArguments("--filter=name=" + containerName);
+ return this;
+ }
+
+ /**
+ * Get the full command.
+ * @return the full command.
+ */
+ @Override
+ public String getCommandWithArguments() {
Review comment:
You are right. I will remove this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm merged pull request #3287: [STORM-3271] enable supervisors to launch a worker inside a docker container
Posted by GitBox <gi...@apache.org>.
Ethanlm merged pull request #3287:
URL: https://github.com/apache/storm/pull/3287
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a change in pull request #3287: [STORM-3271] enable supervisors to launch a worker inside a docker container
Posted by GitBox <gi...@apache.org>.
agresch commented on a change in pull request #3287:
URL: https://github.com/apache/storm/pull/3287#discussion_r450930020
##########
File path: storm-server/src/main/java/org/apache/storm/utils/ServerUtils.java
##########
@@ -290,10 +290,24 @@ public static String scriptFilePath(String dir) {
* @param dir the directory under which the script is to be written
* @param command the command the script is to execute
* @param environment optional environment variables to set before running the script's command. May be null.
+ * @param umask umask to be set. It can be null.
Review comment:
this looks like it should be added to writeScript() below
##########
File path: docs/Docker-support.md
##########
@@ -0,0 +1,135 @@
+---
+title: Docker Support
+layout: documentation
+documentation: true
+---
+
+# Docker Support
+
+This page describes how storm supervisor launches the worker in a docker container.
+
+Note: This is only tested on RHEL7.
+
+## Motivation
+
+This is mostly about security and portability. With workers running inside of docker containers, we isolate running user code from each other and from the hosted machine so that the whole system is less vulnerable to attack.
+It also allows users to run their topologies on different os versions using different docker images.
+
+## Implementation
+
+The implementation is pretty easy to understand. Essentially, `DockerManager` composes a docker-run command and uses `worker-launcher` executable to execute the command
Review comment:
I think this first sentence (easy to understand) is irrelevant. I would drop it. Nice that it is easy though!
##########
File path: docs/Docker-support.md
##########
@@ -0,0 +1,135 @@
+---
+title: Docker Support
+layout: documentation
+documentation: true
+---
+
+# Docker Support
+
+This page describes how storm supervisor launches the worker in a docker container.
+
+Note: This is only tested on RHEL7.
Review comment:
"This has only been tested" maybe?
##########
File path: docs/Docker-support.md
##########
@@ -0,0 +1,135 @@
+---
+title: Docker Support
+layout: documentation
+documentation: true
+---
+
+# Docker Support
+
+This page describes how storm supervisor launches the worker in a docker container.
+
+Note: This is only tested on RHEL7.
+
+## Motivation
+
+This is mostly about security and portability. With workers running inside of docker containers, we isolate running user code from each other and from the hosted machine so that the whole system is less vulnerable to attack.
Review comment:
"This feature is mostly"...?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] kishorvpatil commented on a change in pull request #3287: [STORM-3271] enable supervisors to launch a worker inside a docker container
Posted by GitBox <gi...@apache.org>.
kishorvpatil commented on a change in pull request #3287:
URL: https://github.com/apache/storm/pull/3287#discussion_r449280056
##########
File path: storm-server/src/main/java/org/apache/storm/container/docker/DockerPsCommand.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.container.docker;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.lang.StringUtils;
+
+public class DockerPsCommand extends DockerCommand {
+ private static final String PS_COMMAND = "ps";
+
+ public DockerPsCommand() {
+ super(PS_COMMAND);
+ }
+
+ public DockerPsCommand withQuietOption() {
+ super.addCommandArguments("--quiet=true");
+ return this;
+ }
+
+ public DockerPsCommand withNameFilter(String containerName) {
+ super.addCommandArguments("--filter=name=" + containerName);
+ return this;
+ }
+
+ /**
+ * Get the full command.
+ * @return the full command.
+ */
+ @Override
+ public String getCommandWithArguments() {
Review comment:
Do we need to override this method? Looks like the parent _DockerCommand.getCommandWithArguments_ is doing the same thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org