[jira] [Commented] (HDFS-6881) The DFSClient should use the sampler to determine whether to initiate trace spans when making RPCv9 calls to the NN and DN

["Colin Patrick McCabe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142940#comment-14142940 ] 

Colin Patrick McCabe commented on HDFS-6881:
--------------------------------------------

There are a lot of DistributedFileSystem operations that just involve making an RPC to the NameNode or the DataNode via RPCv9.  We currently never trace those, unless there is already an active trace span.  This patch changes this so that RPCv9 calls will use the HTrace {{Sampler}} configured by the DFSClient to decide whether to initiate a trace span when making an RPC.

This is done by adding a new method to the {{RpcInvocationHandler}} interface, which is implemented by {{ProtobufRpcEngine#Invoker}}.  This new method changes the sampler that the invoker is using.  The {{DFSClient}} uses it to tell the proxy object it creates which HTrace sampler to use.

There a little bit of overlap between this patch and the patch in HDFS-7055 (Add tracing to DFSInputStream), since both patches rely on the {{dfs.client.trace.sampler}} option being present.  But both patches can compile and be committed on their own.

I changed the trace span name format that we're using slightly.  Formerly, we would populate trace spans with names like {{org.apache.hadoop.hdfs.protocol.ClientProtocol.getServerDefaults}}.  This is a rather long name, and I think {{ClientProtocol#getServerDefaults}} is just as descriptive, and unlikely to lead to any confusion.

Requests from HTrace for a configuration key X now cause it to access {{hadoop.htrace.X}} rather than {{hadoop.X}}.  Hopefully, this will avoid potential configuration key name conflicts for any possible HTrace configuration key someone might want to implement.  I also modified the class search to prepend {{org.htrace.impl.}} to class names with no namespace given.  This should lead to shorter configurations, and also protect us against any changes in HTrace in the future (like moving the span receivers to a different namespace).

SpanReceiverHost now sets {{hadoop.htrace.local-file-span-receiver.path}} to {{/tmp/\$PID}} if it is not already set.  This makes it a lot easier to test tracing, since you can just feed all daemons the same config and get nice trace files in {[/tmp}}.

> The DFSClient should use the sampler to determine whether to initiate trace spans when making RPCv9 calls to the NN and DN
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6881
>                 URL: https://issues.apache.org/jira/browse/HDFS-6881
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: 2.6.0
>            Reporter: Masatake Iwasaki
>            Assignee: Colin Patrick McCabe
>
> The DFSClient should use the configred HTrace sampler to determine whether to initiate trace spans when making RPCv9 calls to the NN and DN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)