You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Priyanka B <pr...@gmail.com> on 2015/07/23 17:33:44 UTC
Re: HTrace
Hi Biju and Masatake,
I have a similar problem for my client htrace spans. I am not able to see
the parent client span in my trace output.
I have enabled htrace logging on server side using the below configuration
in hbase-site.xml
<property>
<name>hbase.trace.spanreceiver.classes</name>
<value>org.cloudera.htrace.impl.LocalFileSpanReceiver</value>
</property>
<property>
<name>hbase.local-file-span-receiver.path</name>
<value>/tmp/Server_htrace.out</value>
</property>
On the client side, I maintain a different configuration file
hbase-client.xml
<property>
<name>hbase.trace.spanreceiver.classes</name>
<value>org.cloudera.htrace.impl.LocalFileSpanReceiver</value>
</property>
<property>
<name>hbase.local-file-span-receiver.path</name>
<value>/tmp/Client_htrace.out</value>
</property>
I am able to see the htrace spans when I start/close spans using the hbase
shell so I think server configuration is correct.
However, when I start and close span in my client side code, I see only the
server side span traces without the root parent span that is initiating the
server rpc calls.
Here is how I made changes to the client side code:
this.spanReceiverHost = SpanReceiverHost.getInstance(conf);
I have initialized the spanReceiverHost like above. The conf is
hbase-client.xml here.
Then before a "Get" call I have done a start and close of trace:
TraceScope ts = Trace.startSpan("*Gets*", Sampler.ALWAYS);
try {
result = getTable(table).get(g);
}
catch (IOException e) {
throw new DatabaseException("Error fetching data: "
+ e.getMessage());
}
finally {
if (ts != null) {
ts.close();
}
}
For all the gets I see spans from the server side but no span with "*Gets*"
in the description. I cannot make a graph out of the spans since it is
missing the parent span.
Am I missing some configuration? It will be great if someone could help me
on this. I have one region server, one master running in my environment and
I am using hbase-0.98.8, htrace-20.4.
Thanks a lot,
Priyanka
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073471.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: HTrace
Posted by Nick Dimiduk <nd...@gmail.com>.
FYI Priyanka, hbase-dev@hadoop.apache.org is effectively /dev/null. If
you'd like HBase devs to see your questions, please use dev@hbase.apache.org
instead.
On Tue, Aug 4, 2015 at 11:51 AM, Priyanka B <pr...@gmail.com>
wrote:
> Hi Masatake,
>
> I was not able to get the client traces but I can find a workaround to
> generate those spans myself and get a graph. I had one more question
> related
> to htrace.
>
> If both client process and server use the same file to log htraces, then is
> there a possibility of corruption while writing to the span log file? Since
> there will be 2 JVMs writing to the same file. In addition, our client is
> multithreaded too. I can use synchronize on the client side and make sure
> multiple threads write the client spans in order, however there can be a
> problem if a server span is being written at the same time. Let me know
> what
> you recommend. Thank you.
>
> Best Regards,
> Priyanka
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073682.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Re: HTrace
Posted by Priyanka B <pr...@gmail.com>.
Hi Masatake,
I was not able to get the client traces but I can find a workaround to
generate those spans myself and get a graph. I had one more question related
to htrace.
If both client process and server use the same file to log htraces, then is
there a possibility of corruption while writing to the span log file? Since
there will be 2 JVMs writing to the same file. In addition, our client is
multithreaded too. I can use synchronize on the client side and make sure
multiple threads write the client spans in order, however there can be a
problem if a server span is being written at the same time. Let me know what
you recommend. Thank you.
Best Regards,
Priyanka
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073682.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: HTrace
Posted by Priyanka B <pr...@gmail.com>.
No I do not have multiple processes on the client. I tried a standalone
program with the same configuration and that works fine and generates traces
too. It is only the client code where tomcat is running is unable to
generate spans.
Thanks,
Priyanka
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073524.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: HTrace
Posted by Priyanka Bhalerao-Deshpande <pr...@gmail.com>.
Thank you. I will make sure I use different log files as I had mentioned in
my previous post on the client configuration for HTrace.
Thanks,
Priyanka
On Wed, Aug 5, 2015 at 6:43 PM, Masatake Iwasaki [via Apache HBase] <
ml-node+s679495n4073718h44@n3.nabble.com> wrote:
> Hi Priyanka,
>
> > If both client process and server use the same file to log htraces,
> then is
> > there a possibility of corruption while writing to the span log file?
>
> The client and server are running on the same host?
> You should use different output file for each process.
> It is not safe to write to the same file from multiple processes
> though it is not problem to write spans from multiple threads in single
> process.
>
> Masatake
>
>
> On 8/6/15 01:56, Priyanka Bhalerao-Deshpande wrote:
> > Hi Masatake,
> >
> > I was not able to get the client traces but I can find a workaround to
> > generate those spans myself and get a graph. I had one more question
> > related to htrace.
> >
> > If both client process and server use the same file to log htraces,
> then is
> > there a possibility of corruption while writing to the span log file?
> Since
> > there will be 2 JVMs writing to the same file. In addition, our client
> is
> > multithreaded too. I can use synchronize on the client side and make
> sure
> > multiple threads write the client spans in exclusion, however there
> can be
> > a problem if a server span is being written at the same time. Let me
> know
> > what you recommend. Thank you.
> >
> > Best Regards,
> > Priyanka
> >
> > On Mon, Jul 27, 2015 at 10:51 AM, Masatake Iwasaki [via Apache HBase] <
> > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4073718&i=0>>
> wrote:
> >
> >> > Thank you for getting back to me. Yes I see the Client_htrace.out
> >> getting
> >> > created on the client node. However it is empty. We are using
> >> htrace-2.04
> >> > version. I believe that writes the span asynchronously. Also the
> >> client node
> >> > is running tomcat for serving requests. Would this be a problem?
> >>
> >> Hmm... Do you have multiple processes on the client?
> >> SpanReceiverHost must be initialized in each process.
> >> If you call SpanReceiverHost#getInstance in one process and
> >> call Trace#startSpan in another process,
> >> the client span is not written to file.
> >>
> >> I think running Tomcat is not related to the issue.
> >>
> >>
> >> > Is there any need to call closeReceivers in the client side code ?
> I
> >> tried
> >> > it but that did not seem to work.
> >>
> >> SpanReceiverHost#closeReceivers should be called on just before
> process
> >> exit
> >> but spans will be written to file immediately without that.
> >>
> >>
> >>
> >> On 7/28/15 01:47, Priyanka B wrote:
> >> > Hi Masatake,
> >> >
> >> > Thank you for getting back to me. Yes I see the Client_htrace.out
> >> getting
> >> > created on the client node. However it is empty. We are using
> >> htrace-2.04
> >> > version. I believe that writes the span asynchronously. Also the
> >> client node
> >> > is running tomcat for serving requests. Would this be a problem?
> >> >
> >> > Is there any need to call closeReceivers in the client side code ?
> I
> >> tried
> >> > it but that did not seem to work.
> >> >
> >> >
> >> > Thanks,
> >> > Priyanka
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073515.html
> >> > Sent from the HBase Developer mailing list archive at Nabble.com.
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073518.html
> >> To unsubscribe from HTrace, click here
> >>
> <
> >> .
> >> NAML
> >>
> <
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> >>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073718.html
> To unsubscribe from HTrace, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056705&code=cHJpeWFua2EuYmhhbGVyYW9AZ21haWwuY29tfDQwNTY3MDV8LTEwODc3MjQ5NTQ=>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: HTrace
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Priyanka,
> If both client process and server use the same file to log htraces,
then is
> there a possibility of corruption while writing to the span log file?
The client and server are running on the same host?
You should use different output file for each process.
It is not safe to write to the same file from multiple processes
though it is not problem to write spans from multiple threads in single
process.
Masatake
On 8/6/15 01:56, Priyanka Bhalerao-Deshpande wrote:
> Hi Masatake,
>
> I was not able to get the client traces but I can find a workaround to
> generate those spans myself and get a graph. I had one more question
> related to htrace.
>
> If both client process and server use the same file to log htraces,
then is
> there a possibility of corruption while writing to the span log file?
Since
> there will be 2 JVMs writing to the same file. In addition, our client is
> multithreaded too. I can use synchronize on the client side and make sure
> multiple threads write the client spans in exclusion, however there
can be
> a problem if a server span is being written at the same time. Let me know
> what you recommend. Thank you.
>
> Best Regards,
> Priyanka
>
> On Mon, Jul 27, 2015 at 10:51 AM, Masatake Iwasaki [via Apache HBase] <
> ml-node+s679495n4073518h44@n3.nabble.com> wrote:
>
>> > Thank you for getting back to me. Yes I see the Client_htrace.out
>> getting
>> > created on the client node. However it is empty. We are using
>> htrace-2.04
>> > version. I believe that writes the span asynchronously. Also the
>> client node
>> > is running tomcat for serving requests. Would this be a problem?
>>
>> Hmm... Do you have multiple processes on the client?
>> SpanReceiverHost must be initialized in each process.
>> If you call SpanReceiverHost#getInstance in one process and
>> call Trace#startSpan in another process,
>> the client span is not written to file.
>>
>> I think running Tomcat is not related to the issue.
>>
>>
>> > Is there any need to call closeReceivers in the client side code ? I
>> tried
>> > it but that did not seem to work.
>>
>> SpanReceiverHost#closeReceivers should be called on just before process
>> exit
>> but spans will be written to file immediately without that.
>>
>>
>>
>> On 7/28/15 01:47, Priyanka B wrote:
>> > Hi Masatake,
>> >
>> > Thank you for getting back to me. Yes I see the Client_htrace.out
>> getting
>> > created on the client node. However it is empty. We are using
>> htrace-2.04
>> > version. I believe that writes the span asynchronously. Also the
>> client node
>> > is running tomcat for serving requests. Would this be a problem?
>> >
>> > Is there any need to call closeReceivers in the client side code ? I
>> tried
>> > it but that did not seem to work.
>> >
>> >
>> > Thanks,
>> > Priyanka
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073515.html
>> > Sent from the HBase Developer mailing list archive at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073518.html
>> To unsubscribe from HTrace, click here
>>
<http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056705&code=cHJpeWFua2EuYmhhbGVyYW9AZ21haWwuY29tfDQwNTY3MDV8LTEwODc3MjQ5NTQ=>
>> .
>> NAML
>>
<http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
Re: HTrace
Posted by Priyanka Bhalerao-Deshpande <pr...@gmail.com>.
Hi Masatake,
I was not able to get the client traces but I can find a workaround to
generate those spans myself and get a graph. I had one more question
related to htrace.
If both client process and server use the same file to log htraces, then is
there a possibility of corruption while writing to the span log file? Since
there will be 2 JVMs writing to the same file. In addition, our client is
multithreaded too. I can use synchronize on the client side and make sure
multiple threads write the client spans in exclusion, however there can be
a problem if a server span is being written at the same time. Let me know
what you recommend. Thank you.
Best Regards,
Priyanka
On Mon, Jul 27, 2015 at 10:51 AM, Masatake Iwasaki [via Apache HBase] <
ml-node+s679495n4073518h44@n3.nabble.com> wrote:
> > Thank you for getting back to me. Yes I see the Client_htrace.out
> getting
> > created on the client node. However it is empty. We are using
> htrace-2.04
> > version. I believe that writes the span asynchronously. Also the
> client node
> > is running tomcat for serving requests. Would this be a problem?
>
> Hmm... Do you have multiple processes on the client?
> SpanReceiverHost must be initialized in each process.
> If you call SpanReceiverHost#getInstance in one process and
> call Trace#startSpan in another process,
> the client span is not written to file.
>
> I think running Tomcat is not related to the issue.
>
>
> > Is there any need to call closeReceivers in the client side code ? I
> tried
> > it but that did not seem to work.
>
> SpanReceiverHost#closeReceivers should be called on just before process
> exit
> but spans will be written to file immediately without that.
>
>
>
> On 7/28/15 01:47, Priyanka B wrote:
> > Hi Masatake,
> >
> > Thank you for getting back to me. Yes I see the Client_htrace.out
> getting
> > created on the client node. However it is empty. We are using
> htrace-2.04
> > version. I believe that writes the span asynchronously. Also the
> client node
> > is running tomcat for serving requests. Would this be a problem?
> >
> > Is there any need to call closeReceivers in the client side code ? I
> tried
> > it but that did not seem to work.
> >
> >
> > Thanks,
> > Priyanka
> >
> >
> >
> > --
> > View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073515.html
> > Sent from the HBase Developer mailing list archive at Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073518.html
> To unsubscribe from HTrace, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056705&code=cHJpeWFua2EuYmhhbGVyYW9AZ21haWwuY29tfDQwNTY3MDV8LTEwODc3MjQ5NTQ=>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: HTrace
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> Thank you for getting back to me. Yes I see the Client_htrace.out getting
> created on the client node. However it is empty. We are using htrace-2.04
> version. I believe that writes the span asynchronously. Also the
client node
> is running tomcat for serving requests. Would this be a problem?
Hmm... Do you have multiple processes on the client?
SpanReceiverHost must be initialized in each process.
If you call SpanReceiverHost#getInstance in one process and
call Trace#startSpan in another process,
the client span is not written to file.
I think running Tomcat is not related to the issue.
> Is there any need to call closeReceivers in the client side code ? I
tried
> it but that did not seem to work.
SpanReceiverHost#closeReceivers should be called on just before process exit
but spans will be written to file immediately without that.
On 7/28/15 01:47, Priyanka B wrote:
> Hi Masatake,
>
> Thank you for getting back to me. Yes I see the Client_htrace.out getting
> created on the client node. However it is empty. We are using htrace-2.04
> version. I believe that writes the span asynchronously. Also the
client node
> is running tomcat for serving requests. Would this be a problem?
>
> Is there any need to call closeReceivers in the client side code ? I
tried
> it but that did not seem to work.
>
>
> Thanks,
> Priyanka
>
>
>
> --
> View this message in context:
http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073515.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
Re: HTrace
Posted by Priyanka B <pr...@gmail.com>.
Hi Masatake,
Thank you for getting back to me. Yes I see the Client_htrace.out getting
created on the client node. However it is empty. We are using htrace-2.04
version. I believe that writes the span asynchronously. Also the client node
is running tomcat for serving requests. Would this be a problem?
Is there any need to call closeReceivers in the client side code ? I tried
it but that did not seem to work.
Thanks,
Priyanka
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073515.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: HTrace
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Priyanka,
> <property>
> <name>hbase.trace.spanreceiver.classes</name>
> <value>org.cloudera.htrace.impl.LocalFileSpanReceiver</value>
> </property>
>
> <property>
> <name>hbase.local-file-span-receiver.path</name>
> <value>/tmp/Client_htrace.out</value>
> </property>
Do you have /tmp/Client_htrace.out on the client node?
If span receiver is loaded successfully, the file should be created.
# If you use LocalFileSpanReceiver, client span is written to local file
on client node.
Masatake
On 7/24/15 00:33, Priyanka B wrote:
> Hi Biju and Masatake,
>
> I have a similar problem for my client htrace spans. I am not able to see
> the parent client span in my trace output.
>
> I have enabled htrace logging on server side using the below
configuration
> in hbase-site.xml
>
> <property>
> <name>hbase.trace.spanreceiver.classes</name>
> <value>org.cloudera.htrace.impl.LocalFileSpanReceiver</value>
> </property>
>
> <property>
> <name>hbase.local-file-span-receiver.path</name>
> <value>/tmp/Server_htrace.out</value>
> </property>
>
> On the client side, I maintain a different configuration file
> hbase-client.xml
>
> <property>
> <name>hbase.trace.spanreceiver.classes</name>
> <value>org.cloudera.htrace.impl.LocalFileSpanReceiver</value>
> </property>
>
> <property>
> <name>hbase.local-file-span-receiver.path</name>
> <value>/tmp/Client_htrace.out</value>
> </property>
>
> I am able to see the htrace spans when I start/close spans using the
hbase
> shell so I think server configuration is correct.
> However, when I start and close span in my client side code, I see
only the
> server side span traces without the root parent span that is
initiating the
> server rpc calls.
>
> Here is how I made changes to the client side code:
>
> this.spanReceiverHost = SpanReceiverHost.getInstance(conf);
>
> I have initialized the spanReceiverHost like above. The conf is
> hbase-client.xml here.
>
> Then before a "Get" call I have done a start and close of trace:
>
> TraceScope ts = Trace.startSpan("*Gets*", Sampler.ALWAYS);
> try {
> result = getTable(table).get(g);
> }
> catch (IOException e) {
> throw new DatabaseException("Error fetching data: "
> + e.getMessage());
> }
> finally {
> if (ts != null) {
> ts.close();
> }
> }
>
> For all the gets I see spans from the server side but no span with
"*Gets*"
> in the description. I cannot make a graph out of the spans since it is
> missing the parent span.
>
> Am I missing some configuration? It will be great if someone could
help me
> on this. I have one region server, one master running in my
environment and
> I am using hbase-0.98.8, htrace-20.4.
>
> Thanks a lot,
> Priyanka
>
>
>
>
> --
> View this message in context:
http://apache-hbase.679495.n3.nabble.com/HTrace-tp4056705p4073471.html
> Sent from the HBase Developer mailing list archive at Nabble.com.