You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by 孙长 <sm...@163.com> on 2011/07/01 07:11:17 UTC
hadoop metrics isn't collected into mysql
Hi, I am using chukwa 0.4 , and i start a collector and a agent on two nodes, but i only got system metrics in mysql, no hadoop metrics like jvm, dfs, mapred
dbAdmin.sh is running background, log4j.properties and hadoop-metrics.properties in hadoop conf dir are set according to user guide
no dfs, mapred logs in chukwa/log/metrics dir.
thank you ~
Re: hadoop metrics isn't collected into mysql
Posted by Eric Yang <er...@gmail.com>.
Data should store in hadoop_jvm_*_week, hadoop_mapred_*_week,
hadoop_rpc_*_week.
It looks like the system is working properly. There should be some
data on HDFS in /chukwa/repos that has some data for each data type.
Check if files exists in there.
regards,
Eric
2011/7/2 孙长 <sm...@163.com>:
>
> my hadoop version is 0.20.1
> i telnet localhost 9093 and list, then shows:
> adaptor_8567b8b00a5dc746ccd5b8d179873db1)
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 2110354484
> adaptor_4be0d8af3bc253eef29983e5dbd4f0d5)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309528269962.log
> 150865
> adaptor_b5382452cfdc2ae81bd45d8126804843)
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60
> /usr/bin/iostat -x -k 55 2 4771609
> adaptor_f0b4a4e9d4549f38f633c451e06e290f)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> HadoopLog 0
> /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-jobtracker-hadoop10-131-4-70.log
> 63614063
> adaptor_8890f1df2c09c514450025c0502f473d)
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 129031858
> adaptor_bbdbf9f9ccd112b1d2973aabf4e7c422)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578850828.log
> 18561
> adaptor_5df3de68e136f337c5a46b9bf04637d1)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> HadoopLog 0
> /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-secondarynamenode-hadoop10-131-4-70.log
> 51747
> adaptor_31842d1822977c3c492c725cc8b81bda)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527999599.log
> 380835
> adaptor_0e10bfd94287c9a1c2983c7e103e39f0)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527883608.log
> 1060785
> adaptor_46d675cf8ccd92e245509b19f28dcea8)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309527883619.log
> 952265
> adaptor_1cd9e4134b790becae04026c2fda49ff)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582186581.log
> 76459
> adaptor_5a8f9af70b0e591b034f672f4340371c)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582370136.log
> 74779
> adaptor_fa36c905bcb3701ae4183185ebb8af01)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527883616.log
> 380397
> adaptor_a20e4ffdb7d3fad38732f13fc5aa54a5)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527918692.log
> 801761
> adaptor_72174dc3ec7b8f3871c5d277a0f45125)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582273055.log
> 143061
> adaptor_e24367b38aa7970ea475f67a850fd019)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaAgent 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaAgent.log 366838
> adaptor_1ec2f74ed77d549837b7d5d33db64c7e)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> mapred 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309527999602.log
> 10253607
> adaptor_b74c14fdd101db4a7c9bd7190c230428)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578883752.log
> 18568
> adaptor_16b8b916c39b571a2d38c7ec5bda0a62)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578884932.log
> 31245
> adaptor_4b887b4d345650292baa10290cd34693)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> mapred 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309582370140.log
> 2216270
> adaptor_a2e4104972a498bee2619f6ee0b48efa)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chukwaHttpSender 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaHttpSender.log
> 449552
> adaptor_9532d27cd32d9c64757583ea28b37664)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> HadoopLog 0
> /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-namenode-hadoop10-131-4-70.log
> 353185023
> adaptor_12d05a003a847c3fc1b2c850decbc5bd)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578850821.log
> 46459
> adaptor_d096625c34f55d7d9c5bc9950aa51f04)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309500469601.log
> 242968
> adaptor_ea4c5487a8a14a91da47640303f58eb7)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309578850831.log
> 45630
> adaptor_52e2b02867360ff8b60da5fc4acbcf6e)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582186584.log
> 190797
> adaptor_17e122f11752d4f85bb3771962f4ed0b)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> chunkQueue 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chunkQueue.log 376527
> adaptor_6be04642c8e5f3bf89bb12333baaa678)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527917408.log
> 382148
> adaptor_136ad19fb21c0d398072457a231baba2)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> jvm 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582271839.log
> 75547
> adaptor_b6ac14ec507719e8e01a5e9af8cc0cfc)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582633863.log
> 28576
> adaptor_6dd77c53faf6f9224a4ffdf3cb77796b)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582186572.log
> 202433
> adaptor_ffdb380f632a5efb0aa3532df247502f)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> dfs 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309500469604.log
> 607138
> adaptor_e69999b07d7023e6ba08060c85bd9ad7)
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df
> -l 1437897063
> adaptor_ddad8a9e25eb3e3502d7b4d399207ea8)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> HadoopLog 0
> /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/audit.log
> 172088797
> adaptor_469551943c09f2cda60cbe3965c35614)
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped
> rpc 0
> /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309500469592.log
> 797132
>
> you see, the adaptors are running correctly, so ,which mysql tables will
> save mapred data?
>
> At 2011-07-02 13:04:33,"Eric Yang" <er...@gmail.com> wrote:
>
>>Chukwa 0.4 only works with hadoop 0.20.x. If you are using hadoop
>>0.2, then it probably doesn't work.
>>
>>On the nodes that you see the log files, try:
>>
>>telnet localhost 9093
>>list
>>
>>Check if the jvm and mapred files are being streamed over chukwa agent.
>>
>>If it is not, you might need to restart the mapreduce cluster.
>>
>>regards,
>>Eric
>>
>>2011/7/1 孙长 <sm...@163.com>:
>>> yes, there are rpc and mapred logs in log file
>>>
>>> hadoop-metrics.properties is like this, and hadoop version is 0.2
>>>
>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>> # (the "License"); you may not use this file except in compliance with
>>> # the License. You may obtain a copy of the License at
>>> #
>>> # http://www.apache.org/licenses/LICENSE-2.0
>>> #
>>> # Unless required by applicable law or agreed to in writing, software
>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>> # See the License for the specific language governing permissions and
>>> # limitations under the License.
>>> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> chukwaAgent.period=60
>>> chukwaAgent.directory=/tmp/chukwa/log/metrics/
>>>
>>> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> chukwaHttpSender.period=60
>>> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
>>> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> chunkQueue.period=60
>>> chunkQueue.directory=/tmp/chukwa/log/metrics/
>>> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> dfs.period=60
>>> dfs.directory=/tmp/chukwa/log/metrics/
>>> dfs.uuid=true
>>> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> jvm.period=60
>>> jvm.directory=/tmp/chukwa/log/metrics/
>>> jvm.uuid=true
>>> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> mapred.period=60
>>> mapred.directory=/tmp/chukwa/log/metrics/
>>> mapred.uuid=true
>>> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> rpc.period=60
>>> rpc.directory=/tmp/chukwa/log/metrics/
>>> rpc.uuid=true
>>>
>>> At 2011-07-02 11:39:32,"Eric Yang" <er...@gmail.com> wrote:
>>>
>>>>Check the log directory and see if you see any log file with rpc or
>>>>mapred in the file name?
>>>>What does the hadoop-metrics.properties look like? What version of
>>>>hadoop are you running?
>>>>
>>>>regards,
>>>>Eric
>>>>
>>>>2011/7/1 孙长 <sm...@163.com>:
>>>>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>>>>> into all nodes' hadoop conf dir.
>>>>> Now the problem disapears
>>>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
>>>>> data, why?
>>>>> thank you~
>>>>>
>>>>>
>>>>> At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>>>>>
>>>>>>Looks like mapreduce can not find json.jar in its class path. Which
>>>>>>version of Hadoop are you using?
>>>>>>
>>>>>>regards,
>>>>>>Eric
>>>>>>
>>>>>>2011/7/1 孙长 <sm...@163.com>:
>>>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>>>>>> error:
>>>>>>> task error is as follows:
>>>>>>>
>>>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>> at java.lang.Class.forName(Class.java:169)
>>>>>>> at
>>>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>>>>> at
>>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>>>>> at
>>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>>>
>>>>>>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>>>>>>
>>>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>>>>
>>>>>>>>regards,
>>>>>>>>Eric
>>>>>>>>
>>>>>>>>2011/6/30 孙长 <sm...@163.com>:
>>>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>>>>>> dfs, mapred
>>>>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>>>>>> guide
>>>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>>>>> thank you ~
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>
>
>
hadoop metrics isn't collected into mysql
Posted by 孙长 <sm...@163.com>.
my hadoop version is 0.20.1
i telnet localhost 9093 and list, then shows:
adaptor_8567b8b00a5dc746ccd5b8d179873db1) org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60 /usr/bin/top -b -n 1 -c 2110354484
adaptor_4be0d8af3bc253eef29983e5dbd4f0d5) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309528269962.log 150865
adaptor_b5382452cfdc2ae81bd45d8126804843) org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60 /usr/bin/iostat -x -k 55 2 4771609
adaptor_f0b4a4e9d4549f38f633c451e06e290f) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-jobtracker-hadoop10-131-4-70.log 63614063
adaptor_8890f1df2c09c514450025c0502f473d) org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60 /usr/bin/sar -q -r -n ALL 55 129031858
adaptor_bbdbf9f9ccd112b1d2973aabf4e7c422) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578850828.log 18561
adaptor_5df3de68e136f337c5a46b9bf04637d1) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-secondarynamenode-hadoop10-131-4-70.log 51747
adaptor_31842d1822977c3c492c725cc8b81bda) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527999599.log 380835
adaptor_0e10bfd94287c9a1c2983c7e103e39f0) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527883608.log 1060785
adaptor_46d675cf8ccd92e245509b19f28dcea8) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309527883619.log 952265
adaptor_1cd9e4134b790becae04026c2fda49ff) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582186581.log 76459
adaptor_5a8f9af70b0e591b034f672f4340371c) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582370136.log 74779
adaptor_fa36c905bcb3701ae4183185ebb8af01) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527883616.log 380397
adaptor_a20e4ffdb7d3fad38732f13fc5aa54a5) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527918692.log 801761
adaptor_72174dc3ec7b8f3871c5d277a0f45125) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582273055.log 143061
adaptor_e24367b38aa7970ea475f67a850fd019) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaAgent.log 366838
adaptor_1ec2f74ed77d549837b7d5d33db64c7e) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped mapred 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309527999602.log 10253607
adaptor_b74c14fdd101db4a7c9bd7190c230428) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578883752.log 18568
adaptor_16b8b916c39b571a2d38c7ec5bda0a62) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578884932.log 31245
adaptor_4b887b4d345650292baa10290cd34693) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped mapred 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309582370140.log 2216270
adaptor_a2e4104972a498bee2619f6ee0b48efa) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaHttpSender.log 449552
adaptor_9532d27cd32d9c64757583ea28b37664) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-namenode-hadoop10-131-4-70.log 353185023
adaptor_12d05a003a847c3fc1b2c850decbc5bd) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578850821.log 46459
adaptor_d096625c34f55d7d9c5bc9950aa51f04) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309500469601.log 242968
adaptor_ea4c5487a8a14a91da47640303f58eb7) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309578850831.log 45630
adaptor_52e2b02867360ff8b60da5fc4acbcf6e) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582186584.log 190797
adaptor_17e122f11752d4f85bb3771962f4ed0b) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chunkQueue.log 376527
adaptor_6be04642c8e5f3bf89bb12333baaa678) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527917408.log 382148
adaptor_136ad19fb21c0d398072457a231baba2) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped jvm 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582271839.log 75547
adaptor_b6ac14ec507719e8e01a5e9af8cc0cfc) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582633863.log 28576
adaptor_6dd77c53faf6f9224a4ffdf3cb77796b) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582186572.log 202433
adaptor_ffdb380f632a5efb0aa3532df247502f) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped dfs 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309500469604.log 607138
adaptor_e69999b07d7023e6ba08060c85bd9ad7) org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df -l 1437897063
adaptor_ddad8a9e25eb3e3502d7b4d399207ea8) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped HadoopLog 0 /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/audit.log 172088797
adaptor_469551943c09f2cda60cbe3965c35614) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped rpc 0 /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309500469592.log 797132
you see, the adaptors are running correctly, so ,which mysql tables will save mapred data?
At 2011-07-02 13:04:33,"Eric Yang" <er...@gmail.com> wrote:
>Chukwa 0.4 only works with hadoop 0.20.x. If you are using hadoop
>0.2, then it probably doesn't work.
>
>On the nodes that you see the log files, try:
>
>telnet localhost 9093
>list
>
>Check if the jvm and mapred files are being streamed over chukwa agent.
>
>If it is not, you might need to restart the mapreduce cluster.
>
>regards,
>Eric
>
>2011/7/1 孙长 <sm...@163.com>:
>> yes, there are rpc and mapred logs in log file
>>
>> hadoop-metrics.properties is like this, and hadoop version is 0.2
>>
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chukwaAgent.period=60
>> chukwaAgent.directory=/tmp/chukwa/log/metrics/
>>
>> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chukwaHttpSender.period=60
>> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
>> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> chunkQueue.period=60
>> chunkQueue.directory=/tmp/chukwa/log/metrics/
>> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> dfs.period=60
>> dfs.directory=/tmp/chukwa/log/metrics/
>> dfs.uuid=true
>> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> jvm.period=60
>> jvm.directory=/tmp/chukwa/log/metrics/
>> jvm.uuid=true
>> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> mapred.period=60
>> mapred.directory=/tmp/chukwa/log/metrics/
>> mapred.uuid=true
>> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>> rpc.period=60
>> rpc.directory=/tmp/chukwa/log/metrics/
>> rpc.uuid=true
>>
>> At 2011-07-02 11:39:32,"Eric Yang" <er...@gmail.com> wrote:
>>
>>>Check the log directory and see if you see any log file with rpc or
>>>mapred in the file name?
>>>What does the hadoop-metrics.properties look like? What version of
>>>hadoop are you running?
>>>
>>>regards,
>>>Eric
>>>
>>>2011/7/1 孙长 <sm...@163.com>:
>>>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>>>> into all nodes' hadoop conf dir.
>>>> Now the problem disapears
>>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
>>>> data, why?
>>>> thank you~
>>>>
>>>>
>>>> At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>>>>
>>>>>Looks like mapreduce can not find json.jar in its class path. Which
>>>>>version of Hadoop are you using?
>>>>>
>>>>>regards,
>>>>>Eric
>>>>>
>>>>>2011/7/1 孙长 <sm...@163.com>:
>>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>>>>> error:
>>>>>> task error is as follows:
>>>>>>
>>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>> at java.lang.Class.forName(Class.java:169)
>>>>>> at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>>>> at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>>>> at
>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>>
>>>>>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>>>>>
>>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>>>
>>>>>>>regards,
>>>>>>>Eric
>>>>>>>
>>>>>>>2011/6/30 孙长 <sm...@163.com>:
>>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>>>>> dfs, mapred
>>>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>>>>> guide
>>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>>>> thank you ~
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
Re: Re: Re: Re: hadoop metrics isn't collected into mysql
Posted by Eric Yang <er...@gmail.com>.
Chukwa 0.4 only works with hadoop 0.20.x. If you are using hadoop
0.2, then it probably doesn't work.
On the nodes that you see the log files, try:
telnet localhost 9093
list
Check if the jvm and mapred files are being streamed over chukwa agent.
If it is not, you might need to restart the mapreduce cluster.
regards,
Eric
2011/7/1 孙长 <sm...@163.com>:
> yes, there are rpc and mapred logs in log file
>
> hadoop-metrics.properties is like this, and hadoop version is 0.2
>
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chukwaAgent.period=60
> chukwaAgent.directory=/tmp/chukwa/log/metrics/
>
> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chukwaHttpSender.period=60
> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> chunkQueue.period=60
> chunkQueue.directory=/tmp/chukwa/log/metrics/
> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> dfs.period=60
> dfs.directory=/tmp/chukwa/log/metrics/
> dfs.uuid=true
> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> jvm.period=60
> jvm.directory=/tmp/chukwa/log/metrics/
> jvm.uuid=true
> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> mapred.period=60
> mapred.directory=/tmp/chukwa/log/metrics/
> mapred.uuid=true
> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
> rpc.period=60
> rpc.directory=/tmp/chukwa/log/metrics/
> rpc.uuid=true
>
> At 2011-07-02 11:39:32,"Eric Yang" <er...@gmail.com> wrote:
>
>>Check the log directory and see if you see any log file with rpc or
>>mapred in the file name?
>>What does the hadoop-metrics.properties look like? What version of
>>hadoop are you running?
>>
>>regards,
>>Eric
>>
>>2011/7/1 孙长 <sm...@163.com>:
>>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>>> into all nodes' hadoop conf dir.
>>> Now the problem disapears
>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
>>> data, why?
>>> thank you~
>>>
>>>
>>> At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>>>
>>>>Looks like mapreduce can not find json.jar in its class path. Which
>>>>version of Hadoop are you using?
>>>>
>>>>regards,
>>>>Eric
>>>>
>>>>2011/7/1 孙长 <sm...@163.com>:
>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>>>> error:
>>>>> task error is as follows:
>>>>>
>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:169)
>>>>> at
>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>>> at
>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>>> at
>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>>
>>>>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>>>>
>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>>
>>>>>>regards,
>>>>>>Eric
>>>>>>
>>>>>>2011/6/30 孙长 <sm...@163.com>:
>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>>>> dfs, mapred
>>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>>>> guide
>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>>> thank you ~
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>
>
>
Re:Re: Re: Re: hadoop metrics isn't collected into mysql
Posted by 孙长 <sm...@163.com>.
yes, there are rpc and mapred logs in log file
hadoop-metrics.properties is like this, and hadoop version is 0.2
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
chukwaAgent.period=60
chukwaAgent.directory=/tmp/chukwa/log/metrics/
chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
chukwaHttpSender.period=60
chukwaHttpSender.directory=/tmp/chukwa/log/metrics/
chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
chunkQueue.period=60
chunkQueue.directory=/tmp/chukwa/log/metrics/
dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
dfs.period=60
dfs.directory=/tmp/chukwa/log/metrics/
dfs.uuid=true
jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
jvm.period=60
jvm.directory=/tmp/chukwa/log/metrics/
jvm.uuid=true
mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
mapred.period=60
mapred.directory=/tmp/chukwa/log/metrics/
mapred.uuid=true
rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
rpc.period=60
rpc.directory=/tmp/chukwa/log/metrics/
rpc.uuid=true
At 2011-07-02 11:39:32,"Eric Yang" <er...@gmail.com> wrote:
>Check the log directory and see if you see any log file with rpc or
>mapred in the file name?
>What does the hadoop-metrics.properties look like? What version of
>hadoop are you running?
>
>regards,
>Eric
>
>2011/7/1 孙长 <sm...@163.com>:
>> i know where the problem is , i didn't put log4j and hadoop-metrics file
>> into all nodes' hadoop conf dir.
>> Now the problem disapears
>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
>> data, why?
>> thank you~
>>
>>
>> At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>>
>>>Looks like mapreduce can not find json.jar in its class path. Which
>>>version of Hadoop are you using?
>>>
>>>regards,
>>>Eric
>>>
>>>2011/7/1 孙长 <sm...@163.com>:
>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>>> error:
>>>> task error is as follows:
>>>>
>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>> at java.lang.Class.forName0(Native Method)
>>>> at java.lang.Class.forName(Class.java:169)
>>>> at
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>>> at
>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>>> at
>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>
>>>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>>>
>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>>
>>>>>regards,
>>>>>Eric
>>>>>
>>>>>2011/6/30 孙长 <sm...@163.com>:
>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>>> dfs, mapred
>>>>>> dbAdmin.sh is running background, log4j.properties
>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>>> guide
>>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>>> thank you ~
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
Re: Re: Re: hadoop metrics isn't collected into mysql
Posted by Eric Yang <er...@gmail.com>.
Check the log directory and see if you see any log file with rpc or
mapred in the file name?
What does the hadoop-metrics.properties look like? What version of
hadoop are you running?
regards,
Eric
2011/7/1 孙长 <sm...@163.com>:
> i know where the problem is , i didn't put log4j and hadoop-metrics file
> into all nodes' hadoop conf dir.
> Now the problem disapears
> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc
> data, why?
> thank you~
>
>
> At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>
>>Looks like mapreduce can not find json.jar in its class path. Which
>>version of Hadoop are you using?
>>
>>regards,
>>Eric
>>
>>2011/7/1 孙长 <sm...@163.com>:
>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>>> error:
>>> task error is as follows:
>>>
>>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>>> at
>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>>> at
>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>
>>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>>
>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>>HADOOP_HOME/lib, and restart hadoop?
>>>>
>>>>regards,
>>>>Eric
>>>>
>>>>2011/6/30 孙长 <sm...@163.com>:
>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>>> dfs, mapred
>>>>> dbAdmin.sh is running background, log4j.properties
>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>>> guide
>>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>>> thank you ~
>>>>>
>>>>>
>>>
>>>
>>>
>
>
>
Re:Re: Re: hadoop metrics isn't collected into mysql
Posted by 孙长 <sm...@163.com>.
i know where the problem is , i didn't put log4j and hadoop-metrics file into all nodes' hadoop conf dir.
Now the problem disapears
but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc data, why?
thank you~
At 2011-07-01 22:59:41,"Eric Yang" <er...@gmail.com> wrote:
>Looks like mapreduce can not find json.jar in its class path. Which
>version of Hadoop are you using?
>
>regards,
>Eric
>
>2011/7/1 孙长 <sm...@163.com>:
>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
>> error:
>> task error is as follows:
>>
>> Error: java.lang.ClassNotFoundException: org.json.JSONException
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at
>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
>> at
>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
>> at
>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>>
>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>>HADOOP_HOME/lib, and restart hadoop?
>>>
>>>regards,
>>>Eric
>>>
>>>2011/6/30 孙长 <sm...@163.com>:
>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>>> dfs, mapred
>>>> dbAdmin.sh is running background, log4j.properties
>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>>> guide
>>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>>> thank you ~
>>>>
>>>>
>>
>>
>>
Re: Re: hadoop metrics isn't collected into mysql
Posted by Eric Yang <er...@gmail.com>.
Looks like mapreduce can not find json.jar in its class path. Which
version of Hadoop are you using?
regards,
Eric
2011/7/1 孙长 <sm...@163.com>:
> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to
> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs
> error:
> task error is as follows:
>
> Error: java.lang.ClassNotFoundException: org.json.JSONException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at
> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
> at
> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
> at
> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>
>>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>>HADOOP_HOME/lib, and restart hadoop?
>>
>>regards,
>>Eric
>>
>>2011/6/30 孙长 <sm...@163.com>:
>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>>> dfs, mapred
>>> dbAdmin.sh is running background, log4j.properties
>>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>>> guide
>>> no dfs, mapred logs in chukwa/log/metrics dir.
>>> thank you ~
>>>
>>>
>
>
>
Re:Re: hadoop metrics isn't collected into mysql
Posted by 孙长 <sm...@163.com>.
hi,i have copied chukwa-hadoop-*-client.jar and json.jar toHADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs error:
task error is as follows:
Error: java.lang.ClassNotFoundException: org.json.JSONException
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40)
at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95)
at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
At 2011-07-01 13:17:42,"Eric Yang" <er...@gmail.com> wrote:
>Did you copy chukwa-hadoop-*-client.jar and json.jar to
>HADOOP_HOME/lib, and restart hadoop?
>
>regards,
>Eric
>
>2011/6/30 孙长 <sm...@163.com>:
>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
>> dfs, mapred
>> dbAdmin.sh is running background, log4j.properties
>> and hadoop-metrics.properties in hadoop conf dir are set according to user
>> guide
>> no dfs, mapred logs in chukwa/log/metrics dir.
>> thank you ~
>>
>>
Re: hadoop metrics isn't collected into mysql
Posted by Eric Yang <er...@gmail.com>.
Did you copy chukwa-hadoop-*-client.jar and json.jar to
HADOOP_HOME/lib, and restart hadoop?
regards,
Eric
2011/6/30 孙长 <sm...@163.com>:
> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two
> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm,
> dfs, mapred
> dbAdmin.sh is running background, log4j.properties
> and hadoop-metrics.properties in hadoop conf dir are set according to user
> guide
> no dfs, mapred logs in chukwa/log/metrics dir.
> thank you ~
>
>