[jira] [Commented] (STORM-2862) More flexible logging in multilang (Python, Ruby, JS)

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303483#comment-16303483 ] 

Jungtaek Lim commented on STORM-2862:
-------------------------------------

[~hmcc]
You can extract component name and task ID from thread name (only if it is logged from executor threads) though I agree it is not that easy to build parse format.

The format of executor thread is like Thread-<threadId>-<componentId>-executor<executorId>, and the format of executorId is \[startTaskId endTaskId].

You could construct regex for {code} Thread-[0-9]+-(.+)-executor\[([0-9]+) ([0-9]+)\] {code}

Btw, I'm not sure we can modify the log format, which may be backward incompatible for the cases which need to parse log messages.

> More flexible logging in multilang (Python, Ruby, JS)
> -----------------------------------------------------
>
>                 Key: STORM-2862
>                 URL: https://issues.apache.org/jira/browse/STORM-2862
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-client, storm-multilang
>    Affects Versions: 2.0.0, 1.1.1
>            Reporter: Heather McCartney
>            Assignee: Heather McCartney
>            Priority: Trivial
>              Labels: pull-request-available
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> We're running a Storm topology written in Python, using storm.py from storm-multilang. As well as human-readable logs, the topology is also configured to write JSON logs which are sent to ELK.
> At the moment, when storm-core receives a "log" command, it outputs the pid, component name, and the message it received, like so:
> {{ShellLog pid:<pid>, name:<component name> <message>}}
> The code that does this is (currently) in [ShellBolt line 254|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java#L254] and [ShellSpout line 227|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/spout/ShellSpout.java#L227].
> As well as the pid and component name, it would be great to have the task ID, tuple ID, and the ID of the originating tuple - but this would make parsing the string even more laborious than it is now, and would make the default log message too long. 
> Would it be possible to put contextual information like this in the [ThreadContext|https://logging.apache.org/log4j/2.x/manual/thread-context.html] instead? Then our JSON layout could read from the context instead of parsing the string, and human-readable logs could use "%mdc" in the PatternLayout format string.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)