Shutdown metric question

[DSuiter RDX <ds...@rdx.com>]

Hi all,

We have a TCP source channeled to an Avro sink, which is connecting to an
Avro source channeled to HDFS sink. Everything is flowing, but I noticed
this when it shut down:

Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
channel.event.put.attempt == 471
Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
channel.event.put.success == 471
Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
channel.event.take.attempt == 695
Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
channel.event.take.success == 471

Does the difference between the channel.take.attempt == 695 and
take.success == 471 indicate that my channel dropped data, or does it
indicate that the channel tried to grab from the source but did not get
data from it? There is a period when the Avro RPC call connects where it is
not fully socketed, so maybe the channel is grabbing air? If so, that is
ok. If it indicates data got lost, that is not acceptable and hopefully
someone can help me correct the root cause there.

Thanks!
*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com

Re: Shutdown metric question

[DSuiter RDX <ds...@rdx.com>]

Thanks Hari, that was what I was hoping to hear. Based on the first two
lines, that was what I had expected, but wanted to confirm before trusting
anything of value to the agent recipe as it stands.

Sincerely,
*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com


On Tue, Oct 8, 2013 at 6:20 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:

>  No, it does not mean your channel dropped data. It just means that the
> sink tried to read data 695 times from the channel, but the channel could
> only return 471 - obviously because only that many events were written in
> the first place.
>
>
> Thanks,
> Hari
>
> On Tuesday, October 8, 2013 at 12:03 PM, DSuiter RDX wrote:
>
> Hi all,
>
> We have a TCP source channeled to an Avro sink, which is connecting to an
> Avro source channeled to HDFS sink. Everything is flowing, but I noticed
> this when it shut down:
>
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
> channel.event.put.attempt == 471
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
> channel.event.put.success == 471
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
> channel.event.take.attempt == 695
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel.
> channel.event.take.success == 471
>
> Does the difference between the channel.take.attempt == 695 and
> take.success == 471 indicate that my channel dropped data, or does it
> indicate that the channel tried to grab from the source but did not get
> data from it? There is a period when the Avro RPC call connects where it is
> not fully socketed, so maybe the channel is grabbing air? If so, that is
> ok. If it indicates data got lost, that is not acceptable and hopefully
> someone can help me correct the root cause there.
>
> Thanks!
> *Devin Suiter*
> Jr. Data Solutions Software Engineer
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com
>
>
>

Re: Shutdown metric question

[Hari Shreedharan <hs...@cloudera.com>]

No, it does not mean your channel dropped data. It just means that the sink tried to read data 695 times from the channel, but the channel could only return 471 - obviously because only that many events were written in the first place. 


Thanks,
Hari


On Tuesday, October 8, 2013 at 12:03 PM, DSuiter RDX wrote:

> Hi all,
> 
> We have a TCP source channeled to an Avro sink, which is connecting to an Avro source channeled to HDFS sink. Everything is flowing, but I noticed this when it shut down: 
> 
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel. channel.event.put.attempt == 471
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel. channel.event.put.success == 471
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel. channel.event.take.attempt == 695
> Shutdown Metric for type: CHANNEL, name: memory_RT_Tier2_Channel. channel.event.take.success == 471
> 
> Does the difference between the channel.take.attempt == 695 and take.success == 471 indicate that my channel dropped data, or does it indicate that the channel tried to grab from the source but did not get data from it? There is a period when the Avro RPC call connects where it is not fully socketed, so maybe the channel is grabbing air? If so, that is ok. If it indicates data got lost, that is not acceptable and hopefully someone can help me correct the root cause there. 
> 
> Thanks!
> Devin Suiter
> Jr. Data Solutions Software Engineer
> 
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com (http://www.rdx.com/)