[jira] [Updated] (STORM-158) Live tracing of tuples through the flow

["Rick Kellogg (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rick Kellogg updated STORM-158:
-------------------------------
    Component/s: storm-core

> Live tracing of tuples through the flow
> ---------------------------------------
>
>                 Key: STORM-158
>                 URL: https://issues.apache.org/jira/browse/STORM-158
>             Project: Apache Storm
>          Issue Type: New Feature
>          Components: storm-core
>            Reporter: James Xu
>            Priority: Minor
>
> https://github.com/nathanmarz/storm/issues/531
> Storm should let you bless a record as a "tracer bullet", to be specially reported on as it progresses through the flow. It's important that this be completely transparent -- that is, I can unintrusively switch tracing on in the flow, and that tracer bullets are real, live records (not specially-crafted packets). The intent is that a small fraction of records be tracer bullets. If you harm performance by passing too many through, that's your fault for passing too many through.
> @maphysics is working on code to implement this --- we're finding it very useful for debugging flows -- and so we'd like to see if this is functionality you'd pull into the mainline or storm-contrib. (If this already exists, please advise.)
> Pragmatically, what we're working on is a TracerHook that...
> * In 'prepare', captures some helpful information about the flow.
> * The hook points (emit, boltExecute, both Ack/Fails) look for a field called '_trace'; if absent or null, they return immediately. Otherwise, the field is a HashMap indicating that the tuple should be traced. (We're using a HashMap to allow whoever designated the tuple for tracing to inject extra metadata for the trace report)
> * If it's a tracer bullet each hook point simply writes a verbose, helpfully-formatted biography of the record to the log (the execute hook is more verbose than the others).
> the component_id, sources & targets, etc of the bolt/spout
> the hook point it hit ('emit', 'ack', etc)
> list of output tuple's values in regular order
> * ...and records minimal provenance:
> The execute hook saves off (into a private variable on the hook) the _trace field of the input tuple (a question about this is below)
> calls to the emit hook take the saved _trace info and duplicate it into the output tuple
> Some Questions
> Should these be metrics or hooks? Right now, we're using the hook functionality, not the metrics, because...
> * It wasn't clear how to inspect the tuple from the metric
> * The lifecycle of the metrics matches the bolt's, not the record's -- we'd prefer as-rapid-as-reasonable reporting, tied to the tuple's progress
> * Along with, basically, using metrics would require more spelunking to figure out... We'll follow up to the mailing list with questions on this. So it seems like a hook is the right thing, although cycling a trace trail back to nimbus has a lot of appeal.
> Are the hook points dependably executed in-order? That is, if the execute hook point for bolt A on tuple Q is invoked, can we depend that (until execute is called again), the calls to emit and then to ack/fail are direct consequence of processing tuple Q? (The code seems to say yes, but can we treat that as part of the contract?)
> How do we transparently carry the traceinfo all the way through the topology_ -- yet not annotate every single bolt/spout with trace_info as a field? @maphysics is following up with a separate issue on this. We need to decorate each tuple generated from processing a tracer bullet with a _trace field of its own -- but without modifying the topology or its bolts.
> /cc @maphysics @kornypoet
> As mentioned, we'd eventually like to dispatch tracings to nimbus (or somewhere central). Instead of metrics, another approach would send them to an implicit 'tracings' stream, similar to the 'failure stream' mentioned in #13. Has there been any progress on implicit failure streams?
> see also #146



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)