Flume Monitoring Context

[Ma...@gdc4s.com]

Can anyone shed some light on what each of these metrics are trying to
represent? Some are self explanatory but I'm trying to figure out how I
can use these to troubleshoot my deployment.

 

{

"CHANNEL.fileChannel":{"EventPutSuccessCount":"468085",

                      "Type":"CHANNEL",

                      "StopTime":"0",

                      "EventPutAttemptCount":"468086",

                      "ChannelSize":"233428",

                      "StartTime":"1344882233070",

                      "EventTakeSuccessCount":"458200",

                      "ChannelCapacity":"600000",

                      "EventTakeAttemptCount":"458288"},

"CHANNEL.memChannel":{"EventPutSuccessCount":"22948908",

                   "Type":"CHANNEL",

                   "StopTime":"0",

                   "EventPutAttemptCount":"22948908",

                   "ChannelSize":"5",

                   "StartTime":"1344882209413",

                   "EventTakeSuccessCount":"22948900",

                   "ChannelCapacity":"100",

                   "EventTakeAttemptCount":"22948908"}

}

 

Thanks,

 

Matt 

This message and/or attachments may include information subject to GDC4S
S.P. 1.8.6 and GD Corporate Policy 07-105 and are intended to be
accessed only by authorized recipients.  Use, storage and transmission
are governed by General Dynamics and its policies. Contractual
restrictions apply to third parties.  Recipients should refer to the
policies or contract to determine proper handling.  Unauthorized review,
use, disclosure or distribution is prohibited.  If you are not an
intended recipient, please contact the sender and destroy all copies of
the original message.

 

Re: Flume Monitoring Context

[Roshan Naik <ro...@hortonworks.com>]

Based on mem channel code.. should be similar semantics on file channel...

EventPutAttemptCount : Number of attempts to put events into the
channel. Updated when Source calls Transaction.put(event)

Channel Size - Number of events held in the channel currently

EventTakeSuccessCount : Number of take attempts by sink which actually
succeeded (a non null event was obtained). It is updated when
Transaction.take() is invoked.

EventTakeAttemptCount : Number of times an attempt to Take an event
form channel was made. It is updated when sink commits transaction.

Channel Capacity : The max  number of events that the channel is
configured to hold.


If  EventTakeAttemptCount - EventTakeSuccessCount  < batchSize on sink
side  it is usually an indication that an uncommitted transaction is
in flight on the sink side.

If  EventTakeAttemptCount - EventTakeSuccessCount  > batchSize and you
have only one sink attached to the channel, then the drain rate is
probably faster than the ingest rate. So in a few instance the
Transaction.take() yielded null to the sink.


Changes in ChannelSize is worth keeping an eye on. Increase in channel
size indicates a surge in ingest rate. When it decreases it indicates
a drop in ingest rate.

-Roshan


On Fri, Mar 15, 2013 at 4:54 AM,  <Ma...@gdc4s.com> wrote:
> Can anyone shed some light on what each of these metrics are trying to
> represent? Some are self explanatory but I’m trying to figure out how I can
> use these to troubleshoot my deployment.
>
>
>
> {
>
> "CHANNEL.fileChannel":{"EventPutSuccessCount":"468085",
>
>                       "Type":"CHANNEL",
>
>                       "StopTime":"0",
>
>                       "EventPutAttemptCount":"468086",
>
>                       "ChannelSize":"233428",
>
>                       "StartTime":"1344882233070",
>
>                       "EventTakeSuccessCount":"458200",
>
>                       "ChannelCapacity":"600000",
>
>                       "EventTakeAttemptCount":"458288"},
>
> "CHANNEL.memChannel":{"EventPutSuccessCount":"22948908",
>
>                    "Type":"CHANNEL",
>
>                    "StopTime":"0",
>
>                    "EventPutAttemptCount":"22948908",
>
>                    "ChannelSize":"5",
>
>                    "StartTime":"1344882209413",
>
>                    "EventTakeSuccessCount":"22948900",
>
>                    "ChannelCapacity":"100",
>
>                    "EventTakeAttemptCount":"22948908"}
>
> }
>
>
>
> Thanks,
>
>
>
> Matt
>
> This message and/or attachments may include information subject to GDC4S
> S.P. 1.8.6 and GD Corporate Policy 07-105 and are intended to be accessed
> only by authorized recipients.  Use, storage and transmission are governed
> by General Dynamics and its policies. Contractual restrictions apply to
> third parties.  Recipients should refer to the policies or contract to
> determine proper handling.  Unauthorized review, use, disclosure or
> distribution is prohibited.  If you are not an intended recipient, please
> contact the sender and destroy all copies of the original message.
>
>