[jira] [Commented] (STORM-1342) support multiple logviewers per host for container-isolated worker logs

["Erik Weathers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118650#comment-15118650 ] 

Erik Weathers commented on STORM-1342:
--------------------------------------

STORM-1494 is adding support for the supervisor logs to be linked from the Nimbus UI.  So this will likely be another area to adjust when (if!?) this is fixed.

> support multiple logviewers per host for container-isolated worker logs
> -----------------------------------------------------------------------
>
>                 Key: STORM-1342
>                 URL: https://issues.apache.org/jira/browse/STORM-1342
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>            Reporter: Erik Weathers
>            Priority: Minor
>
> h3. Storm-on-Mesos Worker Logs are in varying directories
> When using [storm-on-mesos|https://github.com/mesos/storm] with cgroups, each topology's workers are isolated into separate containers.  By default the worker logs will be saved into container-specific sandbox directories.  These directories are also topology-specific by definition, because, as just stated, the containers are specific to each topology.
> h3. Problem: Storm supports 1-and-only-1 Logviewer per Worker Host
> A challenge with this different way of running Storm is that the [Storm logviewer|https://github.com/apache/storm/blob/768a85926373355c15cc139fd86268916abc6850/docs/_posts/2013-12-08-storm090-released.md#log-viewer-ui] runs as a single instance on each worker host.   This doesn't play well with having the topology worker logs in separate per-topology containers.  The one logviewer doesn't know about the various sandbox directories that the Storm Workers are writing to.  And if we just spawned new logviewers for each container, the problem is that the Storm UI only knows about 1 global port the logviewer, so you cannot just direct.
> These problems are documented (or linked to) from [Issue #6 in the storm-on-mesos project|https://github.com/mesos/storm/issues/6]
> h3. Possible Solutions I can envision
> # configure the Storm workers to write to log directories that exist on the raw host outside of the container sandbox, and run a single logviewer on a host, which serves up the contents of that directory.
> #* violates one of the basic reasons for using containers: isolation.
> #* also prevents allow a standard use case for Mesos: running more than 1 instance of a Mesos Framework (e.g., "Storm Cluster") at once on same Mesos Cluster. e.g., for Blue-Green deployments.
> #* a variation on this proposal is to somehow expose the sandbox dirs of all storm containers to this singleton logviewer process (still has above problems)
> # launch a separate logviewers in each container, and somehow register those logviewers with Storm such that Storm knows for a given host which logviewer port is assigned to a given topology.
> #* this is the proposed solution
> h3. Storm Changes for the Proposed Solution
> Nimbus or ZooKeeper could serve as a registrar, recording the association between a slot (host + worker port) and the logviewer port that is serving the workers logs. And the Storm-on-Mesos framework could update this registry when launching a new worker.  (This proposal definitely calls for thorough vetting and thinking.)
> h3. Storm-on-Mesos Framework Changes for the Proposed Solution
> Along with the interaction with the "registrar" proposed above, the storm-on-mesos framework can be enhanced to launch multiple logviewers on a given worker host, where each logviewer is dedicated to serving the worker logs from a specific topology's container/sandbox directory.  This would be done by launching a logviewer process within the topology's container, and assigning it an arbitrary listening port that has been determined dynamically through mesos (which treats ports as one of the schedulable resource primitives of a worker host).  [Code implementing this logviewer-port-allocation logic already exists|https://github.com/mesos/storm/commit/af8c49beac04b530c33c1401c829caaa8e368a35], but [that specific portion of the code was reverted|https://github.com/mesos/storm/commit/dc3eee0f0e9c06f6da7b2fe697a8e4fc05b5227e] because of the issues that inspired this ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)