You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yun peng <pe...@gmail.com> on 2014/08/09 17:01:57 UTC
Server-side tracing in HBase (or how deep the tracing is)
Hi,
I am using HTracing-0.3.4 with LocalFileSpanReceiver in HBase-0.99-SNAPSHOT
(in branch1 as of now). I have enabled the tracing configuration (i.e. in
hbase-site.xml) at both client and server side. I am running HBase in a
psuedo-dstributed mode, that is, 2 region servers, one hmaster and one
zookeeper, as different processes but in one machine.
The traces I got is mostly the interactions between zookeeper and
Hbase-client thread.
I am asking how I can enable the trace collection on the server side, like
RPC involving HMaster, and Regionserver. Or does the current hbase-trace
support such server-side tracing? and how deep (thread-level or RPC-level)
the hbase-trace is?
--
Best,
Yun
Re: Server-side tracing in HBase (or how deep the tracing is)
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> In my traces, I did not see any traces from client to either master or
> regionserver. I have tried both deployment, pseudo-distributed on one
> machine, and full-distributed on three machines (one client, one as hmaster
> and zk, and one as regionserver). It only shows the following four spans in
> my trace.
Please check the permission of output file.
Tracing information of the rpc to server is written by the server.
It is also necessary to restart servers after editing hdfs-site.xml
in order to enable tracing.
If you use LocalFileSpanReceiver in distributed cluster,
output files are written to the local of each node.
You should use ZipkinSpanReceiver in distributed cluster.
LocalFileSpanReceiver is for trivial testing.
(8/13/14, 23:04), yun peng wrote:
> In my traces, I did not see any traces from client to either master or
> regionserver. I have tried both deployment, pseudo-distributed on one
> machine, and full-distributed on three machines (one client, one as hmaster
> and zk, and one as regionserver). It only shows the following four spans in
> my trace.
>
> Not sure if it matters, but my hbase cluster (for testing purpose) has only
> one data item. Does it mean the data may only exist on zookeeper?...
>
> ============================
> {"TraceID":7257699385989462904,"SpanID":7302065526441692199,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865229,"Stop":1407995865595,"Description":"RecoverableZookeeper.exists","KVAnnotations":{},"TLAnnotations":[]}
> {"TraceID":7257699385989462904,"SpanID":1531433462825976451,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865598,"Stop":1407995865676,"Description":"RecoverableZookeeper.getData","KVAnnotations":{},"TLAnnotations":[]}
> {"TraceID":7257699385989462904,"SpanID":6733360713454706168,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865955,"Stop":1407995866024,"Description":"RecoverableZookeeper.getData","KVAnnotations":{},"TLAnnotations":[]}
> {"TraceID":7257699385989462904,"SpanID":3479247883930615740,"ParentID":477902,"ProcessID":"Executable","Start":1407995864542,"Stop":1407995866420,"Description":"Gets","KVAnnotations":{},"TLAnnotations":[]}
> ============================
>
>
>
> On Sat, Aug 9, 2014 at 11:48 AM, Masatake Iwasaki <
> iwasakims@oss.nttdata.co.jp> wrote:
>
>>> The traces I got is mostly the interactions between zookeeper and
>>> Hbase-client thread.
>>
>> Can you see at least one trace from client to master or regionserver?
>> If so, it's working correctly.
>>
>>
>>
>>> I am asking how I can enable the trace collection on the server side,
>> like
>>> RPC involving HMaster, and Regionserver. Or does the current hbase-trace
>>> support such server-side tracing? and how deep (thread-level or
>> RPC-level)
>>> the hbase-trace is?
>>
>> Tracing is based on RPC.
>> Downstream RPCs from server processing your traced request to another
>> server are traced.
>> Also you can add custom tracing points.
>> grepping "startSpan" in HBase source code shows you them.
>> Not so many in released server code.
>>
>>
>> (8/9/14, 8:01), yun peng wrote:> Hi,
>>
>>> I am using HTracing-0.3.4 with LocalFileSpanReceiver in
>> HBase-0.99-SNAPSHOT
>>> (in branch1 as of now). I have enabled the tracing configuration (i.e. in
>>> hbase-site.xml) at both client and server side. I am running HBase in a
>>> psuedo-dstributed mode, that is, 2 region servers, one hmaster and one
>>> zookeeper, as different processes but in one machine.
>>>
>>> The traces I got is mostly the interactions between zookeeper and
>>> Hbase-client thread.
>>> I am asking how I can enable the trace collection on the server side,
>> like
>>> RPC involving HMaster, and Regionserver. Or does the current hbase-trace
>>> support such server-side tracing? and how deep (thread-level or
>> RPC-level)
>>> the hbase-trace is?
>>
>
>
>
Re: Server-side tracing in HBase (or how deep the tracing is)
Posted by yun peng <pe...@gmail.com>.
In my traces, I did not see any traces from client to either master or
regionserver. I have tried both deployment, pseudo-distributed on one
machine, and full-distributed on three machines (one client, one as hmaster
and zk, and one as regionserver). It only shows the following four spans in
my trace.
Not sure if it matters, but my hbase cluster (for testing purpose) has only
one data item. Does it mean the data may only exist on zookeeper?...
============================
{"TraceID":7257699385989462904,"SpanID":7302065526441692199,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865229,"Stop":1407995865595,"Description":"RecoverableZookeeper.exists","KVAnnotations":{},"TLAnnotations":[]}
{"TraceID":7257699385989462904,"SpanID":1531433462825976451,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865598,"Stop":1407995865676,"Description":"RecoverableZookeeper.getData","KVAnnotations":{},"TLAnnotations":[]}
{"TraceID":7257699385989462904,"SpanID":6733360713454706168,"ParentID":3479247883930615740,"ProcessID":"Executable","Start":1407995865955,"Stop":1407995866024,"Description":"RecoverableZookeeper.getData","KVAnnotations":{},"TLAnnotations":[]}
{"TraceID":7257699385989462904,"SpanID":3479247883930615740,"ParentID":477902,"ProcessID":"Executable","Start":1407995864542,"Stop":1407995866420,"Description":"Gets","KVAnnotations":{},"TLAnnotations":[]}
============================
On Sat, Aug 9, 2014 at 11:48 AM, Masatake Iwasaki <
iwasakims@oss.nttdata.co.jp> wrote:
> > The traces I got is mostly the interactions between zookeeper and
> > Hbase-client thread.
>
> Can you see at least one trace from client to master or regionserver?
> If so, it's working correctly.
>
>
>
> > I am asking how I can enable the trace collection on the server side,
> like
> > RPC involving HMaster, and Regionserver. Or does the current hbase-trace
> > support such server-side tracing? and how deep (thread-level or
> RPC-level)
> > the hbase-trace is?
>
> Tracing is based on RPC.
> Downstream RPCs from server processing your traced request to another
> server are traced.
> Also you can add custom tracing points.
> grepping "startSpan" in HBase source code shows you them.
> Not so many in released server code.
>
>
> (8/9/14, 8:01), yun peng wrote:> Hi,
>
> > I am using HTracing-0.3.4 with LocalFileSpanReceiver in
> HBase-0.99-SNAPSHOT
> > (in branch1 as of now). I have enabled the tracing configuration (i.e. in
> > hbase-site.xml) at both client and server side. I am running HBase in a
> > psuedo-dstributed mode, that is, 2 region servers, one hmaster and one
> > zookeeper, as different processes but in one machine.
> >
> > The traces I got is mostly the interactions between zookeeper and
> > Hbase-client thread.
> > I am asking how I can enable the trace collection on the server side,
> like
> > RPC involving HMaster, and Regionserver. Or does the current hbase-trace
> > support such server-side tracing? and how deep (thread-level or
> RPC-level)
> > the hbase-trace is?
>
--
Best,
Yun
Re: Server-side tracing in HBase (or how deep the tracing is)
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> The traces I got is mostly the interactions between zookeeper and
> Hbase-client thread.
Can you see at least one trace from client to master or regionserver?
If so, it's working correctly.
> I am asking how I can enable the trace collection on the server side, like
> RPC involving HMaster, and Regionserver. Or does the current hbase-trace
> support such server-side tracing? and how deep (thread-level or RPC-level)
> the hbase-trace is?
Tracing is based on RPC.
Downstream RPCs from server processing your traced request to another server are traced.
Also you can add custom tracing points.
grepping "startSpan" in HBase source code shows you them.
Not so many in released server code.
(8/9/14, 8:01), yun peng wrote:> Hi,
> I am using HTracing-0.3.4 with LocalFileSpanReceiver in HBase-0.99-SNAPSHOT
> (in branch1 as of now). I have enabled the tracing configuration (i.e. in
> hbase-site.xml) at both client and server side. I am running HBase in a
> psuedo-dstributed mode, that is, 2 region servers, one hmaster and one
> zookeeper, as different processes but in one machine.
>
> The traces I got is mostly the interactions between zookeeper and
> Hbase-client thread.
> I am asking how I can enable the trace collection on the server side, like
> RPC involving HMaster, and Regionserver. Or does the current hbase-trace
> support such server-side tracing? and how deep (thread-level or RPC-level)
> the hbase-trace is?