You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Jesse Chang <je...@gmail.com> on 2012/10/03 10:24:42 UTC
Unable to use Swimlanes visualization.
Hi,
I am using Chukwa version 0.5.0 and the CDH4 package - "Version: 4.0.4 (#79
built by jenkins on 20120821-1039 git:
119fe7452f0c356dcde763a1d22eda9363d24ad4)". I re-built Chukwa 0.5.0 with
cdh4 jar files - hbase-0.92.1-cdh4.0.0.jar,
hadoop-common-2.0.0-cdh4.0.1.jar, hadoop-core-2.0.0-mr1-cdh4.0.1.jar, and
hadoop-hdfs-2.0.0-cdh4.0.1.jar.
I could see some data such as shuffle_output_bytes was collected at
http://myhost:4080/hicc/jsp/graph_explorer.jsp. I ran the aggregation
script - share/chukwa/script/pig/ClusterSummary.pig - successfully to see
Cluster Status graphs and MapReduce graphs at http://myhost:4080/hicc/#.
There are no HDFS metrics graphs at the HDFS tab though.
First of all, is the Swimlanes visualization available at Chukwa 0.5.0? If
not, what is the current status?
How can I check if Demux is happening? I can see
org.apache.hadoop.chukwa.extraction.demux.DemuxManager is running, but I
don't see anything is really happening. What would be the cause? How can I
debug it? Is Demux still necessary to see a swimlanes visualization of
Mapreduce stages?
Basically I followed the guide at
http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html and
http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html. I just found
that Chukwa Demux was removed from the picture when I compared it to the
0.4.0 guide at http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html.
Still does Chukwa 0.5.0 support Demux?
What would be the best way to see swimlanes of mapreduce stages?
Thanks,
Jesse
Re: Unable to use Swimlanes visualization.
Posted by Eric Yang <er...@gmail.com>.
Hi Jesse,
Swimlanes and HDFS heatmap are disabled in Chukwa 0.5. Those features were made with mySQL queries. In Chukwa 0.4, data were loaded to mySQL and visualization query mySQL database to generate graphs. In Chukwa 0.5, data storage was converted to HBase but not all data made the transition. There is still a lot of work to convert job data and HDFS client trace to HBase, and update swimlanes and heatmap visualization code to read from HBase. If anyone would like to update those features to work with HBase, this is most welcome.
Sent from my iPhone
On Oct 3, 2012, at 1:24 AM, Jesse Chang <je...@gmail.com> wrote:
> Hi,
>
> I am using Chukwa version 0.5.0 and the CDH4 package - "Version: 4.0.4 (#79 built by jenkins on 20120821-1039 git: 119fe7452f0c356dcde763a1d22eda9363d24ad4)". I re-built Chukwa 0.5.0 with cdh4 jar files - hbase-0.92.1-cdh4.0.0.jar, hadoop-common-2.0.0-cdh4.0.1.jar, hadoop-core-2.0.0-mr1-cdh4.0.1.jar, and hadoop-hdfs-2.0.0-cdh4.0.1.jar.
>
> I could see some data such as shuffle_output_bytes was collected at http://myhost:4080/hicc/jsp/graph_explorer.jsp. I ran the aggregation script - share/chukwa/script/pig/ClusterSummary.pig - successfully to see Cluster Status graphs and MapReduce graphs at http://myhost:4080/hicc/#. There are no HDFS metrics graphs at the HDFS tab though.
>
> First of all, is the Swimlanes visualization available at Chukwa 0.5.0? If not, what is the current status?
>
> How can I check if Demux is happening? I can see org.apache.hadoop.chukwa.extraction.demux.DemuxManager is running, but I don't see anything is really happening. What would be the cause? How can I debug it? Is Demux still necessary to see a swimlanes visualization of Mapreduce stages?
>
> Basically I followed the guide at http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html and http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html. I just found that Chukwa Demux was removed from the picture when I compared it to the 0.4.0 guide at http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html. Still does Chukwa 0.5.0 support Demux?
>
> What would be the best way to see swimlanes of mapreduce stages?
>
> Thanks,
> Jesse
>
>
>
>
>
Re: Unable to use Swimlanes visualization.
Posted by Eric Yang <er...@gmail.com>.
In some implementations of Swimlanes, it doesn't work because of
incompatibilities with Hadoop >= 0.21 due to a change in the
JobHistory log format. The parser may need to be updated for this
to work with newer version of Hadoop.
regards,
Eric
On Sun, Oct 7, 2012 at 3:07 PM, Jesse Chang <je...@gmail.com> wrote:
> I'm getting the following error message from postprocess.log after I
> switched to Chukwa 0.4.0.
>
> 2012-10-07 14:17:41,166 ERROR main FSMDataLoader -
> java.io.FileNotFoundException: File
> /chukwa/temp/fsm_1349644589617_org.apache.hadoop.chukwa.analysis.salsa.fsm.JobHistoryTaskDataMapper
> does not exist.
>
> I got the same kind of error messages when I used Chukwa 0.5.0. What am I
> missing?
>
> Thanks,
> Jesse
>
> On Wed, Oct 3, 2012 at 2:05 PM, Jesse Chang <je...@gmail.com> wrote:
>>
>> OK. I added org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter
>> to chukwaCollector.pipeline, and I am getting some data such as
>> /chukwa/postProcess/demuxOutputDir_1349295946720/MyCluster/mapred_tasktracker.
>>
>> But it cannot process data further because of the following errors.
>>
>> 2012-10-03 13:10:45,023 ERROR pool-1-thread-1 MetricDataLoader -
>> Initialization failed for: MyCluster. Please check jdbc configuration.
>>
>> I guess I need MySQL and follow the guide for the version 0.4.
>>
>> Jesse
>>
>>
>> On Wed, Oct 3, 2012 at 1:24 AM, Jesse Chang <je...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I am using Chukwa version 0.5.0 and the CDH4 package - "Version: 4.0.4
>>> (#79 built by jenkins on 20120821-1039 git:
>>> 119fe7452f0c356dcde763a1d22eda9363d24ad4)". I re-built Chukwa 0.5.0 with
>>> cdh4 jar files - hbase-0.92.1-cdh4.0.0.jar,
>>> hadoop-common-2.0.0-cdh4.0.1.jar, hadoop-core-2.0.0-mr1-cdh4.0.1.jar, and
>>> hadoop-hdfs-2.0.0-cdh4.0.1.jar.
>>>
>>> I could see some data such as shuffle_output_bytes was collected at
>>> http://myhost:4080/hicc/jsp/graph_explorer.jsp. I ran the aggregation script
>>> - share/chukwa/script/pig/ClusterSummary.pig - successfully to see Cluster
>>> Status graphs and MapReduce graphs at http://myhost:4080/hicc/#. There are
>>> no HDFS metrics graphs at the HDFS tab though.
>>>
>>> First of all, is the Swimlanes visualization available at Chukwa 0.5.0?
>>> If not, what is the current status?
>>>
>>> How can I check if Demux is happening? I can see
>>> org.apache.hadoop.chukwa.extraction.demux.DemuxManager is running, but I
>>> don't see anything is really happening. What would be the cause? How can I
>>> debug it? Is Demux still necessary to see a swimlanes visualization of
>>> Mapreduce stages?
>>>
>>> Basically I followed the guide at
>>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html and
>>> http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html. I just found that
>>> Chukwa Demux was removed from the picture when I compared it to the 0.4.0
>>> guide at http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html. Still
>>> does Chukwa 0.5.0 support Demux?
>>>
>>> What would be the best way to see swimlanes of mapreduce stages?
>>>
>>> Thanks,
>>> Jesse
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Unable to use Swimlanes visualization.
Posted by Jesse Chang <je...@gmail.com>.
I'm getting the following error message from postprocess.log after I
switched to Chukwa 0.4.0.
2012-10-07 14:17:41,166 ERROR main FSMDataLoader -
java.io.FileNotFoundException: File
/chukwa/temp/fsm_1349644589617_org.apache.hadoop.chukwa.analysis.salsa.fsm.JobHistoryTaskDataMapper
does not exist.
I got the same kind of error messages when I used Chukwa 0.5.0. What am I
missing?
Thanks,
Jesse
On Wed, Oct 3, 2012 at 2:05 PM, Jesse Chang <je...@gmail.com> wrote:
> OK. I added org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter
> to chukwaCollector.pipeline, and I am getting some data such as
> /chukwa/postProcess/demuxOutputDir_1349295946720/MyCluster/mapred_tasktracker.
>
> But it cannot process data further because of the following errors.
>
> 2012-10-03 13:10:45,023 ERROR pool-1-thread-1 MetricDataLoader -
> Initialization failed for: MyCluster. Please check jdbc configuration.
>
> I guess I need MySQL and follow the guide for the version 0.4.
>
> Jesse
>
>
> On Wed, Oct 3, 2012 at 1:24 AM, Jesse Chang <je...@gmail.com>wrote:
>
>> Hi,
>>
>> I am using Chukwa version 0.5.0 and the CDH4 package - "Version: 4.0.4
>> (#79 built by jenkins on 20120821-1039 git:
>> 119fe7452f0c356dcde763a1d22eda9363d24ad4)". I re-built Chukwa 0.5.0 with
>> cdh4 jar files - hbase-0.92.1-cdh4.0.0.jar,
>> hadoop-common-2.0.0-cdh4.0.1.jar, hadoop-core-2.0.0-mr1-cdh4.0.1.jar, and
>> hadoop-hdfs-2.0.0-cdh4.0.1.jar.
>>
>> I could see some data such as shuffle_output_bytes was collected at
>> http://myhost:4080/hicc/jsp/graph_explorer.jsp. I ran the aggregation
>> script - share/chukwa/script/pig/ClusterSummary.pig - successfully to see
>> Cluster Status graphs and MapReduce graphs at http://myhost:4080/hicc/#.
>> There are no HDFS metrics graphs at the HDFS tab though.
>>
>> First of all, is the Swimlanes visualization available at Chukwa 0.5.0?
>> If not, what is the current status?
>>
>> How can I check if Demux is happening? I can see
>> org.apache.hadoop.chukwa.extraction.demux.DemuxManager is running, but I
>> don't see anything is really happening. What would be the cause? How can I
>> debug it? Is Demux still necessary to see a swimlanes visualization of
>> Mapreduce stages?
>>
>> Basically I followed the guide at
>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.htmland
>> http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html. I just found
>> that Chukwa Demux was removed from the picture when I compared it to the
>> 0.4.0 guide at http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html.
>> Still does Chukwa 0.5.0 support Demux?
>>
>> What would be the best way to see swimlanes of mapreduce stages?
>>
>> Thanks,
>> Jesse
>>
>>
>>
>>
>>
>>
>
Re: Unable to use Swimlanes visualization.
Posted by Jesse Chang <je...@gmail.com>.
OK. I added org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter
to chukwaCollector.pipeline, and I am getting some data such as
/chukwa/postProcess/demuxOutputDir_1349295946720/MyCluster/mapred_tasktracker.
But it cannot process data further because of the following errors.
2012-10-03 13:10:45,023 ERROR pool-1-thread-1 MetricDataLoader -
Initialization failed for: MyCluster. Please check jdbc configuration.
I guess I need MySQL and follow the guide for the version 0.4.
Jesse
On Wed, Oct 3, 2012 at 1:24 AM, Jesse Chang <je...@gmail.com> wrote:
> Hi,
>
> I am using Chukwa version 0.5.0 and the CDH4 package - "Version: 4.0.4
> (#79 built by jenkins on 20120821-1039 git:
> 119fe7452f0c356dcde763a1d22eda9363d24ad4)". I re-built Chukwa 0.5.0 with
> cdh4 jar files - hbase-0.92.1-cdh4.0.0.jar,
> hadoop-common-2.0.0-cdh4.0.1.jar, hadoop-core-2.0.0-mr1-cdh4.0.1.jar, and
> hadoop-hdfs-2.0.0-cdh4.0.1.jar.
>
> I could see some data such as shuffle_output_bytes was collected at
> http://myhost:4080/hicc/jsp/graph_explorer.jsp. I ran the aggregation
> script - share/chukwa/script/pig/ClusterSummary.pig - successfully to see
> Cluster Status graphs and MapReduce graphs at http://myhost:4080/hicc/#.
> There are no HDFS metrics graphs at the HDFS tab though.
>
> First of all, is the Swimlanes visualization available at Chukwa 0.5.0? If
> not, what is the current status?
>
> How can I check if Demux is happening? I can see
> org.apache.hadoop.chukwa.extraction.demux.DemuxManager is running, but I
> don't see anything is really happening. What would be the cause? How can I
> debug it? Is Demux still necessary to see a swimlanes visualization of
> Mapreduce stages?
>
> Basically I followed the guide at
> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html and
> http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html. I just found
> that Chukwa Demux was removed from the picture when I compared it to the
> 0.4.0 guide at http://incubator.apache.org/chukwa/docs/r0.4.0/admin.html.
> Still does Chukwa 0.5.0 support Demux?
>
> What would be the best way to see swimlanes of mapreduce stages?
>
> Thanks,
> Jesse
>
>
>
>
>
>