[jira] Commented: (HADOOP-4049) Cross-system causal tracing within Hadoop

["Ari Rabkin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627768#action_12627768 ] 

Ari Rabkin commented on HADOOP-4049:
------------------------------------

I spent a little while looking at this and exploring options a month or so ago.

The main conclusion from that work is that it's worth solving this problem generally, and sharing work with other instrumentation tools.   At the points where you'd like to emit causal log entries, there's other things you'd like to do, so probably the right way to do this is to have an abstract "Instrumentation" class, that can be subclassed to do useful things like send x-trace reports.  

There are already instrumentation classes for JobTracker and TaskTracker (via HADOOP-3772), though you may find it necessary to add more methods.  Also, there was some concern about the performance impact of tracing in the IPC level, since it's on the critical path.

> Cross-system causal tracing within Hadoop
> -----------------------------------------
>
>                 Key: HADOOP-4049
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4049
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs, ipc, mapred
>            Reporter: George Porter
>         Attachments: multiblockread.png, multiblockwrite.png
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Much of Hadoop's behavior is client-driven, with clients responsible for contacting individual datanodes to read and write data, as well as dividing up work for map and reduce tasks.  In a large deployment with many concurrent users, identifying the effects of individual clients on the infrastructure is a challenge.  The use of data pipelining in HDFS and Map/Reduce make it hard to follow the effects of a given client request through the system.
> This proposal is to instrument the HDFS, IPC, and Map/Reduce layers of Hadoop with X-Trace.  X-Trace is an open-source framework for capturing causality of events in a distributed system.  It can correlate operations making up a single user request, even if those operations span multiple machines.  As an example, you could use X-Trace to follow an HDFS write operation as it is pipelined through intermediate nodes.  Additionally, you could trace a single Map/Reduce job and see how it is decomposed into lower-layer HDFS operations.
> Matei Zaharia and Andy Konwinski initially integrated X-Trace with a local copy of the 0.14 release, and I've brought that code up to release 0.17.  Performing the integration involves modifying the IPC protocol, inter-datanode protocol, and some data structures in the map/reduce layer to include 20-byte long tracing metadata.  With release 0.18, the generated traces could be collected with Chukwa.
> I've attached some example traces of HDFS and IPC layers from the 0.17 patch to this JIRA issue.
> More information about X-Trace is available from http://www.x-trace.net/ as well as in a paper that appeared at NSDI 2007, available online at http://www.usenix.org/events/nsdi07/tech/fonseca.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.