You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Alexandru Calin <al...@gmail.com> on 2016/08/27 13:52:57 UTC
Tracing Hadoop using HTrace with Zipkin
favorite
<http://stackoverflow.com/questions/39181880/tracing-hadoop-using-htrace-with-zipkin#>
I am trying to use HTrace with Hadoop 2.6.0 on Ubuntu 14.04. I have
followed Hadoop's HTrace integration tutorial here
<https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Tracing.html>
.
I have added the following configuration properties to core-site.xml on all
nodes ( namenode & datanodes ), as specified in the tutorial:
<property>
<name>hadoop.htrace.sampler</name>
<value>NeverSampler</value>
</property>
property>
<name>hadoop.htrace.spanreceiver.classes</name>
<value>ZipkinSpanReceiver</value>
</property>
<property>
<name>hadoop.htrace.zipkin.collector-hostname</name>
<value>hadoop-master</value>
</property>
<property>
<name>hadoop.htrace.zipkin.collector-port</name>
<value>9410</value>
</property>
I have followed the *Setting up ZipkinSpanReceiver* section, compiled,
built & copied the libraries:
/usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar
/usr/local/hadoop/share/hadoop/common/lib/htrace-zipkin-3.0.4-jar-with-dependencies.jar
I have downloaded Zipkin from here <https://github.com/openzipkin/zipkin> and
started the server running the jar.
I have compiled, the TracingFsShell.java, with some minor modifications to
including dependencies:
import org.apache.hadoop.conf.Configuration;import
org.apache.hadoop.fs.FsShell;import
org.apache.hadoop.tracing.SpanReceiverHost;import
org.apache.hadoop.util.ToolRunner;import org.htrace.Sampler;import
org.htrace.Trace;import org.htrace.TraceScope;
public class TracingFsShell {
public static void main(String argv[]) throws Exception {
Configuration conf = new Configuration();
FsShell shell = new FsShell();
conf.setQuietMode(false);
shell.setConf(conf);
SpanReceiverHost.getInstance(conf);
int res = 0;
TraceScope ts = null;
try {
ts = Trace.startSpan("FsShell", Sampler.ALWAYS);
res = ToolRunner.run(shell, argv);
} finally {
shell.close();
if (ts != null) ts.close();
}
System.exit(res);
}}
The modifications came after I wasn't able to build the example and have
looked through the classes of the two compiled jars(htrace-core &
htrace-zipkin). When I run the example, I get nothing in zipkin.
java -cp .:`hadoop classpath` TracingFsShell -ls /Found 1 items
drwxr-xr-x - hduser supergroup 0 2016-07-02 08:28 /user
Am I missing something?
Re: Tracing Hadoop using HTrace with Zipkin
Posted by Alexandru Calin <al...@gmail.com>.
So, for everyone else out there, the thing is that there is a certain
tutorial to be followed for 2.6.0 version of Hadoop, and it can be found
here :
https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Tracing.html
That solves the problem.
On Sat, Aug 27, 2016 at 5:18 PM, Alexandru Calin <alexandrucalin29@gmail.com
> wrote:
> Yes, you are right, sorry, that was I copy-paste mistake form the website,
> haven't noticed it. In reality I am using AlwawsSampler, in my
> configuration, 3 slaves, 1 namenode, lxc containers.
>
> On Sat, Aug 27, 2016 at 5:05 PM, Sandeep Khurana <sk...@gmail.com>
> wrote:
>
>> Never used any of these. But reading documentation of the links you sent "NeverSampler:
>> HTrace is OFF for all spans" . Should you not use either ProbabilitySampler
>> or AlwaysSampler
>>
>> On Sat, Aug 27, 2016 at 7:22 PM, Alexandru Calin <
>> alexandrucalin29@gmail.com> wrote:
>>
>>>
>>> favorite
>>> <http://stackoverflow.com/questions/39181880/tracing-hadoop-using-htrace-with-zipkin#>
>>>
>>> I am trying to use HTrace with Hadoop 2.6.0 on Ubuntu 14.04. I have
>>> followed Hadoop's HTrace integration tutorial here
>>> <https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Tracing.html>
>>> .
>>>
>>> I have added the following configuration properties to core-site.xml on
>>> all nodes ( namenode & datanodes ), as specified in the tutorial:
>>>
>>> <property>
>>> <name>hadoop.htrace.sampler</name>
>>> <value>NeverSampler</value>
>>> </property>
>>> property>
>>> <name>hadoop.htrace.spanreceiver.classes</name>
>>> <value>ZipkinSpanReceiver</value>
>>> </property>
>>> <property>
>>> <name>hadoop.htrace.zipkin.collector-hostname</name>
>>> <value>hadoop-master</value>
>>> </property>
>>> <property>
>>> <name>hadoop.htrace.zipkin.collector-port</name>
>>> <value>9410</value>
>>> </property>
>>>
>>> I have followed the *Setting up ZipkinSpanReceiver* section, compiled,
>>> built & copied the libraries:
>>>
>>> /usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar /usr/local/hadoop/share/hadoop/common/lib/htrace-zipkin-3.0.4-jar-with-dependencies.jar
>>>
>>> I have downloaded Zipkin from here
>>> <https://github.com/openzipkin/zipkin> and started the server running
>>> the jar.
>>>
>>> I have compiled, the TracingFsShell.java, with some minor modifications
>>> to including dependencies:
>>>
>>> import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FsShell;import org.apache.hadoop.tracing.SpanReceiverHost;import org.apache.hadoop.util.ToolRunner;import org.htrace.Sampler;import org.htrace.Trace;import org.htrace.TraceScope;
>>> public class TracingFsShell {
>>> public static void main(String argv[]) throws Exception {
>>> Configuration conf = new Configuration();
>>> FsShell shell = new FsShell();
>>> conf.setQuietMode(false);
>>> shell.setConf(conf);
>>> SpanReceiverHost.getInstance(conf);
>>> int res = 0;
>>> TraceScope ts = null;
>>> try {
>>> ts = Trace.startSpan("FsShell", Sampler.ALWAYS);
>>> res = ToolRunner.run(shell, argv);
>>> } finally {
>>> shell.close();
>>> if (ts != null) ts.close();
>>> }
>>> System.exit(res);
>>> }}
>>>
>>> The modifications came after I wasn't able to build the example and have
>>> looked through the classes of the two compiled jars(htrace-core &
>>> htrace-zipkin). When I run the example, I get nothing in zipkin.
>>>
>>> java -cp .:`hadoop classpath` TracingFsShell -ls /Found 1 items
>>> drwxr-xr-x - hduser supergroup 0 2016-07-02 08:28 /user
>>>
>>> Am I missing something?
>>>
>>
>>
>>
>> --
>> Thanks and regards
>> Sandeep Khurana
>>
>
>
Re: Tracing Hadoop using HTrace with Zipkin
Posted by Alexandru Calin <al...@gmail.com>.
Yes, you are right, sorry, that was I copy-paste mistake form the website,
haven't noticed it. In reality I am using AlwawsSampler, in my
configuration, 3 slaves, 1 namenode, lxc containers.
On Sat, Aug 27, 2016 at 5:05 PM, Sandeep Khurana <sk...@gmail.com>
wrote:
> Never used any of these. But reading documentation of the links you sent "NeverSampler:
> HTrace is OFF for all spans" . Should you not use either ProbabilitySampler
> or AlwaysSampler
>
> On Sat, Aug 27, 2016 at 7:22 PM, Alexandru Calin <
> alexandrucalin29@gmail.com> wrote:
>
>>
>> favorite
>> <http://stackoverflow.com/questions/39181880/tracing-hadoop-using-htrace-with-zipkin#>
>>
>> I am trying to use HTrace with Hadoop 2.6.0 on Ubuntu 14.04. I have
>> followed Hadoop's HTrace integration tutorial here
>> <https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Tracing.html>
>> .
>>
>> I have added the following configuration properties to core-site.xml on
>> all nodes ( namenode & datanodes ), as specified in the tutorial:
>>
>> <property>
>> <name>hadoop.htrace.sampler</name>
>> <value>NeverSampler</value>
>> </property>
>> property>
>> <name>hadoop.htrace.spanreceiver.classes</name>
>> <value>ZipkinSpanReceiver</value>
>> </property>
>> <property>
>> <name>hadoop.htrace.zipkin.collector-hostname</name>
>> <value>hadoop-master</value>
>> </property>
>> <property>
>> <name>hadoop.htrace.zipkin.collector-port</name>
>> <value>9410</value>
>> </property>
>>
>> I have followed the *Setting up ZipkinSpanReceiver* section, compiled,
>> built & copied the libraries:
>>
>> /usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar /usr/local/hadoop/share/hadoop/common/lib/htrace-zipkin-3.0.4-jar-with-dependencies.jar
>>
>> I have downloaded Zipkin from here <https://github.com/openzipkin/zipkin> and
>> started the server running the jar.
>>
>> I have compiled, the TracingFsShell.java, with some minor modifications
>> to including dependencies:
>>
>> import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FsShell;import org.apache.hadoop.tracing.SpanReceiverHost;import org.apache.hadoop.util.ToolRunner;import org.htrace.Sampler;import org.htrace.Trace;import org.htrace.TraceScope;
>> public class TracingFsShell {
>> public static void main(String argv[]) throws Exception {
>> Configuration conf = new Configuration();
>> FsShell shell = new FsShell();
>> conf.setQuietMode(false);
>> shell.setConf(conf);
>> SpanReceiverHost.getInstance(conf);
>> int res = 0;
>> TraceScope ts = null;
>> try {
>> ts = Trace.startSpan("FsShell", Sampler.ALWAYS);
>> res = ToolRunner.run(shell, argv);
>> } finally {
>> shell.close();
>> if (ts != null) ts.close();
>> }
>> System.exit(res);
>> }}
>>
>> The modifications came after I wasn't able to build the example and have
>> looked through the classes of the two compiled jars(htrace-core &
>> htrace-zipkin). When I run the example, I get nothing in zipkin.
>>
>> java -cp .:`hadoop classpath` TracingFsShell -ls /Found 1 items
>> drwxr-xr-x - hduser supergroup 0 2016-07-02 08:28 /user
>>
>> Am I missing something?
>>
>
>
>
> --
> Thanks and regards
> Sandeep Khurana
>
Re: Tracing Hadoop using HTrace with Zipkin
Posted by Sandeep Khurana <sk...@gmail.com>.
Never used any of these. But reading documentation of the links you
sent "NeverSampler:
HTrace is OFF for all spans" . Should you not use either ProbabilitySampler
or AlwaysSampler
On Sat, Aug 27, 2016 at 7:22 PM, Alexandru Calin <alexandrucalin29@gmail.com
> wrote:
>
> favorite
> <http://stackoverflow.com/questions/39181880/tracing-hadoop-using-htrace-with-zipkin#>
>
> I am trying to use HTrace with Hadoop 2.6.0 on Ubuntu 14.04. I have
> followed Hadoop's HTrace integration tutorial here
> <https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Tracing.html>
> .
>
> I have added the following configuration properties to core-site.xml on
> all nodes ( namenode & datanodes ), as specified in the tutorial:
>
> <property>
> <name>hadoop.htrace.sampler</name>
> <value>NeverSampler</value>
> </property>
> property>
> <name>hadoop.htrace.spanreceiver.classes</name>
> <value>ZipkinSpanReceiver</value>
> </property>
> <property>
> <name>hadoop.htrace.zipkin.collector-hostname</name>
> <value>hadoop-master</value>
> </property>
> <property>
> <name>hadoop.htrace.zipkin.collector-port</name>
> <value>9410</value>
> </property>
>
> I have followed the *Setting up ZipkinSpanReceiver* section, compiled,
> built & copied the libraries:
>
> /usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar /usr/local/hadoop/share/hadoop/common/lib/htrace-zipkin-3.0.4-jar-with-dependencies.jar
>
> I have downloaded Zipkin from here <https://github.com/openzipkin/zipkin> and
> started the server running the jar.
>
> I have compiled, the TracingFsShell.java, with some minor modifications to
> including dependencies:
>
> import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FsShell;import org.apache.hadoop.tracing.SpanReceiverHost;import org.apache.hadoop.util.ToolRunner;import org.htrace.Sampler;import org.htrace.Trace;import org.htrace.TraceScope;
> public class TracingFsShell {
> public static void main(String argv[]) throws Exception {
> Configuration conf = new Configuration();
> FsShell shell = new FsShell();
> conf.setQuietMode(false);
> shell.setConf(conf);
> SpanReceiverHost.getInstance(conf);
> int res = 0;
> TraceScope ts = null;
> try {
> ts = Trace.startSpan("FsShell", Sampler.ALWAYS);
> res = ToolRunner.run(shell, argv);
> } finally {
> shell.close();
> if (ts != null) ts.close();
> }
> System.exit(res);
> }}
>
> The modifications came after I wasn't able to build the example and have
> looked through the classes of the two compiled jars(htrace-core &
> htrace-zipkin). When I run the example, I get nothing in zipkin.
>
> java -cp .:`hadoop classpath` TracingFsShell -ls /Found 1 items
> drwxr-xr-x - hduser supergroup 0 2016-07-02 08:28 /user
>
> Am I missing something?
>
--
Thanks and regards
Sandeep Khurana