[jira] [Commented] (STORM-2191) shorten classpaths in worker and LogWriter commands

["Robert Joseph Evans (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991201#comment-15991201 ] 

Robert Joseph Evans commented on STORM-2191:
--------------------------------------------

[~erikdw],

#2060 removes clojure from the worker classpath and along with that kryo-shaded (by way of carbonite).  So now if you want to use the clojure API you need to include the dependency storm-clojure with you topology and it will ship clojure and carbonite, etc.  So it will still show up on your classpath, but we know the order of the classpath and it should not be an issue.

#2060 did not fix the problem for storm-server, as storm-server still has clojure as a dependency for the ui and the logviewer, but once those are moved over to java then there will be no issues with it any longer on master.  

> shorten classpaths in worker and LogWriter commands
> ---------------------------------------------------
>
>                 Key: STORM-2191
>                 URL: https://issues.apache.org/jira/browse/STORM-2191
>             Project: Apache Storm
>          Issue Type: Task
>          Components: storm-core
>    Affects Versions: 1.0.2
>            Reporter: Erik Weathers
>            Priority: Minor
>              Labels: cli, command-line
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> When launching the worker daemon and its wrapping LogWriter daemon, the commands can become so long that they eclipse the default Linux limit of 4096 bytes. That results in commands that are cut off in {{ps}} output, and prevents easily inspecting the system to see even what processes are running.
> The specific scenario in which this problem can be easily triggered: *running Storm on Mesos*.
> h5. Details on why it happens:
> # using the default Mesos containerizer instead of Docker containers, which causes the storm-mesos package to be unpacked into the Mesos executor sandbox.
> # The ["expand all jars on classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140] functionality in the {{bin/storm.py}} script causes every one of the jars that storm bundles into its lib directory to be explicitly listed in the command.
> #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}}
> #* and say that the original classpath argument in the supervisor cmd includes the following for the {{lib/}} dir in the binary storm package:
> #** {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}}
> #* That leads to a hugely expanded classpath argument for the LogWriter and Worker daemons that get launched:
> #** {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)