You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Stuti Awasthi <St...@persistent.co.in> on 2010/10/01 15:15:34 UTC
How to use metadata sent by agent to collector
Hi all,
I noticed that when agent sends chunks to collector it also sends the metadata along with it, but in demux.java class , we are just dealing with the record entry that comes from agent.
How can I use this metadata information that is sent by agent to collector also on collector side.
Example , each chunk sends the meta data given as below , how can I use other fields like Key or capp on my collector side.
{
"DataType": "logtype",
"Key": "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
"Timestamp": 1285917710741,
"mapFields": {
"capp": "/home/hadoop/xend.log",
"csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
"ctags": " cluster=\"chukwa",
"body" : "xyz"
}
Any pointers will be helpful.
Regards
Stuti Awasthi
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: How to use metadata sent by agent to collector
Posted by Bill Graham <bi...@gmail.com>.
No problem. Also note that you should be able to get the same fields
from the ChukwaRecord object if you have access to it, which would
probably be easier.
On Fri, Oct 1, 2010 at 10:52 AM, Stuti Awasthi
<St...@persistent.co.in> wrote:
> Thanks Bill, I will try using chunk object.
>
> -----Original Message-----
> From: Bill Graham [mailto:billgraham@gmail.com]
> Sent: Friday, October 01, 2010 10:34 PM
> To: Stuti Awasthi
> Cc: chukwa-user@incubator.apache.org
> Subject: Re: How to use metadata sent by agent to collector
>
> capp is chunk.streamName
> csource is chunk.source
> ctags is chunk.tags
>
> I believe body is chunk.data but note that a chunk can hold data from
> multiple records so you'd need to check chunk.recordOffsets to know
> how to split it up into multiple records.
>
>
> On Fri, Oct 1, 2010 at 9:32 AM, Stuti Awasthi
> <St...@persistent.co.in> wrote:
>> Hi Bill,
>>
>> I wanted to access the "maps : " fileds
>> "mapFields": {
>>
>> "capp": "/home/hadoop/xend.log",
>> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>> "ctags": " cluster=\"chukwa",
>> "body" : "xyz"
>> }
>>
>> I checked that ChunkImpl class sets all these values. I am looking into the correct code? If yes, I can use relevant key value pair from here.
>>
>> Please Suggest
>> Regards
>> Stuti
>>
>> -----Original Message-----
>> From: Bill Graham [mailto:billgraham@gmail.com]
>> Sent: Friday, October 01, 2010 9:55 PM
>> To: chukwa-user@incubator.apache.org
>> Subject: Re: How to use metadata sent by agent to collector
>>
>> Can you be more specific regarding what you're trying to do? The
>> ServlerCollector, Demux and Processor classes all have access to the
>> Chunk object which encapsulated the fields you list, so it's not clear
>> what field access you're missing.
>>
>> On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
>> <St...@persistent.co.in> wrote:
>>> Hi all,
>>>
>>>
>>>
>>> I noticed that when agent sends chunks to collector it also sends the
>>> metadata along with it, but in demux.java class , we are just dealing with
>>> the record entry that comes from agent.
>>>
>>> How can I use this metadata information that is sent by agent to collector
>>> also on collector side.
>>>
>>>
>>>
>>> Example , each chunk sends the meta data given as below , how can I use
>>> other fields like Key or capp on my collector side.
>>>
>>> {
>>>
>>> "DataType": "logtype",
>>>
>>> "Key":
>>> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>>>
>>> "Timestamp": 1285917710741,
>>>
>>> "mapFields": {
>>>
>>> "capp": "/home/hadoop/xend.log",
>>>
>>> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>>>
>>> "ctags": " cluster=\"chukwa",
>>>
>>> "body" : "xyz"
>>>
>>> }
>>>
>>>
>>>
>>> Any pointers will be helpful.
>>>
>>>
>>>
>>> Regards
>>>
>>> Stuti Awasthi
>>>
>>> DISCLAIMER ========== This e-mail may contain privileged and confidential
>>> information which is the property of Persistent Systems Ltd. It is intended
>>> only for the use of the individual or entity to which it is addressed. If
>>> you are not the intended recipient, you are not authorized to read, retain,
>>> copy, print, distribute or use this message. If you have received this
>>> communication in error, please notify the sender and delete all copies of
>>> this message. Persistent Systems Ltd. does not accept any liability for
>>> virus infected mails.
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
RE: How to use metadata sent by agent to collector
Posted by Stuti Awasthi <St...@persistent.co.in>.
Thanks Bill, I will try using chunk object.
-----Original Message-----
From: Bill Graham [mailto:billgraham@gmail.com]
Sent: Friday, October 01, 2010 10:34 PM
To: Stuti Awasthi
Cc: chukwa-user@incubator.apache.org
Subject: Re: How to use metadata sent by agent to collector
capp is chunk.streamName
csource is chunk.source
ctags is chunk.tags
I believe body is chunk.data but note that a chunk can hold data from
multiple records so you'd need to check chunk.recordOffsets to know
how to split it up into multiple records.
On Fri, Oct 1, 2010 at 9:32 AM, Stuti Awasthi
<St...@persistent.co.in> wrote:
> Hi Bill,
>
> I wanted to access the "maps : " fileds
> "mapFields": {
>
> "capp": "/home/hadoop/xend.log",
> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
> "ctags": " cluster=\"chukwa",
> "body" : "xyz"
> }
>
> I checked that ChunkImpl class sets all these values. I am looking into the correct code? If yes, I can use relevant key value pair from here.
>
> Please Suggest
> Regards
> Stuti
>
> -----Original Message-----
> From: Bill Graham [mailto:billgraham@gmail.com]
> Sent: Friday, October 01, 2010 9:55 PM
> To: chukwa-user@incubator.apache.org
> Subject: Re: How to use metadata sent by agent to collector
>
> Can you be more specific regarding what you're trying to do? The
> ServlerCollector, Demux and Processor classes all have access to the
> Chunk object which encapsulated the fields you list, so it's not clear
> what field access you're missing.
>
> On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
> <St...@persistent.co.in> wrote:
>> Hi all,
>>
>>
>>
>> I noticed that when agent sends chunks to collector it also sends the
>> metadata along with it, but in demux.java class , we are just dealing with
>> the record entry that comes from agent.
>>
>> How can I use this metadata information that is sent by agent to collector
>> also on collector side.
>>
>>
>>
>> Example , each chunk sends the meta data given as below , how can I use
>> other fields like Key or capp on my collector side.
>>
>> {
>>
>> "DataType": "logtype",
>>
>> "Key":
>> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>>
>> "Timestamp": 1285917710741,
>>
>> "mapFields": {
>>
>> "capp": "/home/hadoop/xend.log",
>>
>> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>>
>> "ctags": " cluster=\"chukwa",
>>
>> "body" : "xyz"
>>
>> }
>>
>>
>>
>> Any pointers will be helpful.
>>
>>
>>
>> Regards
>>
>> Stuti Awasthi
>>
>> DISCLAIMER ========== This e-mail may contain privileged and confidential
>> information which is the property of Persistent Systems Ltd. It is intended
>> only for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient, you are not authorized to read, retain,
>> copy, print, distribute or use this message. If you have received this
>> communication in error, please notify the sender and delete all copies of
>> this message. Persistent Systems Ltd. does not accept any liability for
>> virus infected mails.
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: How to use metadata sent by agent to collector
Posted by Bill Graham <bi...@gmail.com>.
capp is chunk.streamName
csource is chunk.source
ctags is chunk.tags
I believe body is chunk.data but note that a chunk can hold data from
multiple records so you'd need to check chunk.recordOffsets to know
how to split it up into multiple records.
On Fri, Oct 1, 2010 at 9:32 AM, Stuti Awasthi
<St...@persistent.co.in> wrote:
> Hi Bill,
>
> I wanted to access the "maps : " fileds
> "mapFields": {
>
> "capp": "/home/hadoop/xend.log",
> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
> "ctags": " cluster=\"chukwa",
> "body" : "xyz"
> }
>
> I checked that ChunkImpl class sets all these values. I am looking into the correct code? If yes, I can use relevant key value pair from here.
>
> Please Suggest
> Regards
> Stuti
>
> -----Original Message-----
> From: Bill Graham [mailto:billgraham@gmail.com]
> Sent: Friday, October 01, 2010 9:55 PM
> To: chukwa-user@incubator.apache.org
> Subject: Re: How to use metadata sent by agent to collector
>
> Can you be more specific regarding what you're trying to do? The
> ServlerCollector, Demux and Processor classes all have access to the
> Chunk object which encapsulated the fields you list, so it's not clear
> what field access you're missing.
>
> On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
> <St...@persistent.co.in> wrote:
>> Hi all,
>>
>>
>>
>> I noticed that when agent sends chunks to collector it also sends the
>> metadata along with it, but in demux.java class , we are just dealing with
>> the record entry that comes from agent.
>>
>> How can I use this metadata information that is sent by agent to collector
>> also on collector side.
>>
>>
>>
>> Example , each chunk sends the meta data given as below , how can I use
>> other fields like Key or capp on my collector side.
>>
>> {
>>
>> "DataType": "logtype",
>>
>> "Key":
>> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>>
>> "Timestamp": 1285917710741,
>>
>> "mapFields": {
>>
>> "capp": "/home/hadoop/xend.log",
>>
>> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>>
>> "ctags": " cluster=\"chukwa",
>>
>> "body" : "xyz"
>>
>> }
>>
>>
>>
>> Any pointers will be helpful.
>>
>>
>>
>> Regards
>>
>> Stuti Awasthi
>>
>> DISCLAIMER ========== This e-mail may contain privileged and confidential
>> information which is the property of Persistent Systems Ltd. It is intended
>> only for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient, you are not authorized to read, retain,
>> copy, print, distribute or use this message. If you have received this
>> communication in error, please notify the sender and delete all copies of
>> this message. Persistent Systems Ltd. does not accept any liability for
>> virus infected mails.
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
RE: How to use metadata sent by agent to collector
Posted by Stuti Awasthi <St...@persistent.co.in>.
Hi Bill,
I wanted to access the "maps : " fileds
"mapFields": {
"capp": "/home/hadoop/xend.log",
"csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
"ctags": " cluster=\"chukwa",
"body" : "xyz"
}
I checked that ChunkImpl class sets all these values. I am looking into the correct code? If yes, I can use relevant key value pair from here.
Please Suggest
Regards
Stuti
-----Original Message-----
From: Bill Graham [mailto:billgraham@gmail.com]
Sent: Friday, October 01, 2010 9:55 PM
To: chukwa-user@incubator.apache.org
Subject: Re: How to use metadata sent by agent to collector
Can you be more specific regarding what you're trying to do? The
ServlerCollector, Demux and Processor classes all have access to the
Chunk object which encapsulated the fields you list, so it's not clear
what field access you're missing.
On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
<St...@persistent.co.in> wrote:
> Hi all,
>
>
>
> I noticed that when agent sends chunks to collector it also sends the
> metadata along with it, but in demux.java class , we are just dealing with
> the record entry that comes from agent.
>
> How can I use this metadata information that is sent by agent to collector
> also on collector side.
>
>
>
> Example , each chunk sends the meta data given as below , how can I use
> other fields like Key or capp on my collector side.
>
> {
>
> "DataType": "logtype",
>
> "Key":
> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>
> "Timestamp": 1285917710741,
>
> "mapFields": {
>
> "capp": "/home/hadoop/xend.log",
>
> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>
> "ctags": " cluster=\"chukwa",
>
> "body" : "xyz"
>
> }
>
>
>
> Any pointers will be helpful.
>
>
>
> Regards
>
> Stuti Awasthi
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: How to use metadata sent by agent to collector
Posted by Bill Graham <bi...@gmail.com>.
Can you be more specific regarding what you're trying to do? The
ServlerCollector, Demux and Processor classes all have access to the
Chunk object which encapsulated the fields you list, so it's not clear
what field access you're missing.
On Fri, Oct 1, 2010 at 6:15 AM, Stuti Awasthi
<St...@persistent.co.in> wrote:
> Hi all,
>
>
>
> I noticed that when agent sends chunks to collector it also sends the
> metadata along with it, but in demux.java class , we are just dealing with
> the record entry that comes from agent.
>
> How can I use this metadata information that is sent by agent to collector
> also on collector side.
>
>
>
> Example , each chunk sends the meta data given as below , how can I use
> other fields like Key or capp on my collector side.
>
> {
>
> "DataType": "logtype",
>
> "Key":
> "1285916400000/iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com/1285917710741",
>
> "Timestamp": 1285917710741,
>
> "mapFields": {
>
> "capp": "/home/hadoop/xend.log",
>
> "csource": "iaas-1-0-16-3e-45-46-7d.iaas.torolab.ibm.com",
>
> "ctags": " cluster=\"chukwa”,
>
> “body” : “xyz”
>
> }
>
>
>
> Any pointers will be helpful.
>
>
>
> Regards
>
> Stuti Awasthi
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.