You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Ve...@cognizant.com on 2010/11/19 07:59:55 UTC
Problem in collecting data from Agent
Hi All,
When I try to send a log file from Agent to Collector, I didn't get
anything in HDFS expect following two file,(it also have no data inside
that)
Please let me know, how to monitor wheather collector is receiving
data..
$ ./hadoop fs -ls /chukwa/logs/
Found 2 items
-rw-r--r-- 3 hadoop supergroup 0 2010-11-19 17:14
/chukwa/logs/201019171406118_HADOOPSQOOP_4b2656712c63c0713a7fea.chukwa
-rw-r--r-- 3 hadoop supergroup 103 2010-11-19 17:25
/chukwa/logs/201019172022791_HADOOPSQOOP_1d40ead912c63f88f598000.done
I was trying to send Iostat.log file which is given in
chukwa/test/samples directory.
Using the adapter
add filetailer.FileTailingAdaptor logs /tmp/Iostat.log 0
OK add completed; new ID is 4b08ea7f0635fff5d4de778f6782bd1b
Collector.log Output:
2010-11-19 17:44:50,418 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
2010-11-19 17:45:18,183 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - start Date [Fri Nov
19 17:45:19 IST 2010]
2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - Rotate from Timer-2
2010-11-19 17:45:19,247 INFO Timer-2 LocalWriter - no chunks written to
/tmp/chukwa/dataSink/201019174019240_HADOOPSQOOP_4c320ac712c63fd13837ffa
.chukwa, deleting
2010-11-19 17:45:20,419 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
2010-11-19 17:45:50,420 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
2010-11-19 17:46:18,184 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2010-11-19 17:46:20,420 INFO Timer-3 LocalWriter -
stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
Agent Console Output:
.......
data type was logs
data length was 131072, not printing
-------------------
Console out connector got event at offset 1441792
data type was logs
data length was 131072, not printing
-------------------
Console out connector got event at offset 1572864
data type was logs
data length was 131072, not printing
-------------------
Console out connector got event at offset 1703936
data type was logs
data length was 131072, not printing
-------------------
Console out connector got event at offset 1740827
data type was logs
data length was 36891, not printing
Thanks & Regards
Venkat
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly prohibited and may be unlawful.
RE: Problem in collecting data from Agent
Posted by Stuti Awasthi <St...@persistent.co.in>.
Hi,
Also check your Http Connection between agent and collector. When collector received the record, it also sends acknowledgement to agent back. You can also see the agents logs to check the issue.
-----Original Message-----
From: Ariel Rabkin [mailto:asrabkin@gmail.com]
Sent: Friday, November 19, 2010 12:37 PM
To: chukwa-user@incubator.apache.org
Subject: Re: Problem in collecting data from Agent
You apparently have your agent set to use the console connector. This
means that the agent is writing to standard out, not to HDFS. Did you
specify "local" when starting the agent?
--Ari
On Thu, Nov 18, 2010 at 10:59 PM, <Ve...@cognizant.com> wrote:
>
> Hi All,
>
>
>
> When I try to send a log file from Agent to Collector, I didn't get anything in HDFS expect following two file,(it also have no data inside that)
>
> Please let me know, how to monitor wheather collector is receiving data..
>
>
>
> $ ./hadoop fs -ls /chukwa/logs/
>
> Found 2 items
>
> -rw-r--r-- 3 hadoop supergroup 0 2010-11-19 17:14 /chukwa/logs/201019171406118_HADOOPSQOOP_4b2656712c63c0713a7fea.chukwa
>
> -rw-r--r-- 3 hadoop supergroup 103 2010-11-19 17:25 /chukwa/logs/201019172022791_HADOOPSQOOP_1d40ead912c63f88f598000.done
>
>
>
> I was trying to send Iostat.log file which is given in chukwa/test/samples directory.
>
>
>
>
>
> Using the adapter
>
>
>
> add filetailer.FileTailingAdaptor logs /tmp/Iostat.log 0
>
> OK add completed; new ID is 4b08ea7f0635fff5d4de778f6782bd1b
>
>
>
> Collector.log Output:
>
> 2010-11-19 17:44:50,418 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:45:18,183 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
>
> 2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - start Date [Fri Nov 19 17:45:19 IST 2010]
>
> 2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - Rotate from Timer-2
>
> 2010-11-19 17:45:19,247 INFO Timer-2 LocalWriter - no chunks written to /tmp/chukwa/dataSink/201019174019240_HADOOPSQOOP_4c320ac712c63fd13837ffa.chukwa, deleting
>
> 2010-11-19 17:45:20,419 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:45:50,420 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:46:18,184 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
>
> 2010-11-19 17:46:20,420 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
>
>
> Agent Console Output:
>
> ...
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1441792
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1572864
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1703936
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1740827
>
> data type was logs
>
> data length was 36891, not printing
>
>
>
> Thanks & Regards
>
> Venkat
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Problem in collecting data from Agent
Posted by Ariel Rabkin <as...@gmail.com>.
You apparently have your agent set to use the console connector. This
means that the agent is writing to standard out, not to HDFS. Did you
specify "local" when starting the agent?
--Ari
On Thu, Nov 18, 2010 at 10:59 PM, <Ve...@cognizant.com> wrote:
>
> Hi All,
>
>
>
> When I try to send a log file from Agent to Collector, I didn’t get anything in HDFS expect following two file,(it also have no data inside that)
>
> Please let me know, how to monitor wheather collector is receiving data..
>
>
>
> $ ./hadoop fs -ls /chukwa/logs/
>
> Found 2 items
>
> -rw-r--r-- 3 hadoop supergroup 0 2010-11-19 17:14 /chukwa/logs/201019171406118_HADOOPSQOOP_4b2656712c63c0713a7fea.chukwa
>
> -rw-r--r-- 3 hadoop supergroup 103 2010-11-19 17:25 /chukwa/logs/201019172022791_HADOOPSQOOP_1d40ead912c63f88f598000.done
>
>
>
> I was trying to send Iostat.log file which is given in chukwa/test/samples directory.
>
>
>
>
>
> Using the adapter
>
>
>
> add filetailer.FileTailingAdaptor logs /tmp/Iostat.log 0
>
> OK add completed; new ID is 4b08ea7f0635fff5d4de778f6782bd1b
>
>
>
> Collector.log Output:
>
> 2010-11-19 17:44:50,418 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:45:18,183 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
>
> 2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - start Date [Fri Nov 19 17:45:19 IST 2010]
>
> 2010-11-19 17:45:19,243 INFO Timer-2 LocalWriter - Rotate from Timer-2
>
> 2010-11-19 17:45:19,247 INFO Timer-2 LocalWriter - no chunks written to /tmp/chukwa/dataSink/201019174019240_HADOOPSQOOP_4c320ac712c63fd13837ffa.chukwa, deleting
>
> 2010-11-19 17:45:20,419 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:45:50,420 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
> 2010-11-19 17:46:18,184 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
>
> 2010-11-19 17:46:20,420 INFO Timer-3 LocalWriter - stat:datacollection.writer.local.LocalWriter dataSize=0 dataRate=0
>
>
>
> Agent Console Output:
>
> …….
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1441792
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1572864
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1703936
>
> data type was logs
>
> data length was 131072, not printing
>
> -------------------
>
> Console out connector got event at offset 1740827
>
> data type was logs
>
> data length was 36891, not printing
>
>
>
> Thanks & Regards
>
> Venkat
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department