You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Buntu Dev <bu...@gmail.com> on 2015/11/02 23:33:22 UTC
Determine the backlog in File channel
I got a File channel with HDFS sink. In the case when the sink slows down
and event taken from the channel falls behind while the event puts continue
at the same pace, how would one go about finding the amount of backlog or
time it takes to clear the backlog?
Thanks!
Re: Determine the backlog in File channel
Posted by Buntu Dev <bu...@gmail.com>.
Thanks for the info Mike and would be great if you can mention the pipeline
to get the flume metrics into TSDB.
I actually wanted to know how to compute the throughput based on the
various metrics that Flume provides:
{
"CHANNEL.my-file-channel": {
"EventPutSuccessCount": "6500",
"ChannelFillPercentage": "0.0",
"Type": "CHANNEL",
"StopTime": "0",
"EventPutAttemptCount": "6500",
"ChannelSize": "0",
"StartTime": "1443053580229",
"EventTakeSuccessCount": "6500",
"ChannelCapacity": "1000000",
"EventTakeAttemptCount": "164715352"
}
Thanks!
On Tue, Nov 3, 2015 at 5:53 AM, Keane, Mike <mk...@conversantmedia.com>
wrote:
> We ping our flume metrics every 5 minutes and push the results to a TSDB (
> http://opentsdb.net/) database. TSDB is great for visualizing your data
> rates. Depending on your flume configuration, you will get greatly varying
> rates. If you are using spinning disks with a file channel you'll want to
> make sure your checkpoint directory and and data directory are on
> physically different disks.
>
>
> ------------------------------
> *From:* Buntu Dev [buntudev@gmail.com]
> *Sent:* Monday, November 02, 2015 7:09 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Determine the backlog in File channel
>
> Awesome thanks!
>
> Whats the right way to calculate the throughput of the channel?
>
> On Mon, Nov 2, 2015 at 4:10 PM, Ahmed Vila <av...@devlogic.eu> wrote:
>
>> To add to it, here is the user manual reference:
>> https://flume.apache.org/FlumeUserGuide.html#json-reporting
>>
>> On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mk...@conversantmedia.com>
>> wrote:
>>
>>> Enable monitoring and you can get the channelFillPercentage and
>>> channelSize from the metrics:
>>>
>>> VM arguments for http monitoring:
>>> -Dflume.monitoring.type=http -Dflume.monitoring.port=12345
>>>
>>> For JMX
>>> -Dcom.sun.management.jmxremote.port=12346
>>>
>>> or whatever port you choose.
>>>
>>>
>>> ------------------------------
>>> *From:* Buntu Dev [buntudev@gmail.com]
>>> *Sent:* Monday, November 02, 2015 4:33 PM
>>> *To:* user@flume.apache.org
>>> *Subject:* Determine the backlog in File channel
>>>
>>> I got a File channel with HDFS sink. In the case when the sink slows
>>> down and event taken from the channel falls behind while the event puts
>>> continue at the same pace, how would one go about finding the amount of
>>> backlog or time it takes to clear the backlog?
>>>
>>> Thanks!
>>>
>>> This email and any files included with it may contain privileged,
>>> proprietary and/or confidential information that is for the sole use
>>> of the intended recipient(s). Any disclosure, copying, distribution,
>>> posting, or use of the information contained in or attached to this
>>> email is prohibited unless permitted by the sender. If you have
>>> received this email in error, please immediately notify the sender
>>> via return email, telephone, or fax and destroy this original
>>> transmission
>>> and its included files without reading or saving it in any manner.
>>> Thank you.
>>>
>>
>>
>>
>> --
>>
>> Best regards,
>> Ahmed Vila | Senior software developer
>> DevLogic | Sarajevo | Bosnia and Herzegovina
>>
>> Office : +387 33 942 123
>> Mobile: +387 62 139 348
>>
>> Website: www.devlogic.eu
>> E-mail : avila@devlogic.eu
>> ---------------------------------------------------------------------
>> This e-mail and any attachment is for authorised use by the intended
>> recipient(s) only. This email contains confidential information. It should
>> not be copied, disclosed to, retained or used by, any party other than the
>> intended recipient. Any unauthorised distribution, dissemination or copying
>> of this E-mail or its attachments, and/or any use of any information
>> contained in them, is strictly prohibited and may be illegal. If you are
>> not an intended recipient then please promptly delete this e-mail and any
>> attachment and all copies and inform the sender directly via email. Any
>> emails that you send to us may be monitored by systems or persons other
>> than the named communicant for the purposes of ascertaining whether the
>> communication complies with the law and company policies.
>>
>> ---------------------------------------------------------------------
>> This e-mail and any attachment is for authorised use by the intended
>> recipient(s) only. This email contains confidential information. It should
>> not be copied, disclosed to, retained or used by, any party other than the
>> intended recipient. Any unauthorised distribution, dissemination or copying
>> of this E-mail or its attachments, and/or any use of any information
>> contained in them, is strictly prohibited and may be illegal. If you are
>> not an intended recipient then please promptly delete this e-mail and any
>> attachment and all copies and inform the sender directly via email. Any
>> emails that you send to us may be monitored by systems or persons other
>> than the named communicant for the purposes of ascertaining whether the
>> communication complies with the law and company policies.
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>
RE: Determine the backlog in File channel
Posted by "Keane, Mike" <mk...@conversantmedia.com>.
We ping our flume metrics every 5 minutes and push the results to a TSDB (http://opentsdb.net/) database. TSDB is great for visualizing your data rates. Depending on your flume configuration, you will get greatly varying rates. If you are using spinning disks with a file channel you'll want to make sure your checkpoint directory and and data directory are on physically different disks.
________________________________
From: Buntu Dev [buntudev@gmail.com]
Sent: Monday, November 02, 2015 7:09 PM
To: user@flume.apache.org
Subject: Re: Determine the backlog in File channel
Awesome thanks!
Whats the right way to calculate the throughput of the channel?
On Mon, Nov 2, 2015 at 4:10 PM, Ahmed Vila <av...@devlogic.eu>> wrote:
To add to it, here is the user manual reference:
https://flume.apache.org/FlumeUserGuide.html#json-reporting
On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mk...@conversantmedia.com>> wrote:
Enable monitoring and you can get the channelFillPercentage and channelSize from the metrics:
VM arguments for http monitoring:
-Dflume.monitoring.type=http -Dflume.monitoring.port=12345
For JMX
-Dcom.sun.management.jmxremote.port=12346
or whatever port you choose.
________________________________
From: Buntu Dev [buntudev@gmail.com<ma...@gmail.com>]
Sent: Monday, November 02, 2015 4:33 PM
To: user@flume.apache.org<ma...@flume.apache.org>
Subject: Determine the backlog in File channel
I got a File channel with HDFS sink. In the case when the sink slows down and event taken from the channel falls behind while the event puts continue at the same pace, how would one go about finding the amount of backlog or time it takes to clear the backlog?
Thanks!
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
--
Best regards,
Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina
Office : +387 33 942 123<tel:%2B387%2033%20942%20123>
Mobile: +387 62 139 348<tel:%2B387%2062%20139%20348>
Website: www.devlogic.eu<http://www.devlogic.eu>
E-mail : avila@devlogic.eu<ma...@devlogic.eu>
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This email contains confidential information. It should not be copied, disclosed to, retained or used by, any party other than the intended recipient. Any unauthorised distribution, dissemination or copying of this E-mail or its attachments, and/or any use of any information contained in them, is strictly prohibited and may be illegal. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender directly via email. Any emails that you send to us may be monitored by systems or persons other than the named communicant for the purposes of ascertaining whether the communication complies with the law and company policies.
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This email contains confidential information. It should not be copied, disclosed to, retained or used by, any party other than the intended recipient. Any unauthorised distribution, dissemination or copying of this E-mail or its attachments, and/or any use of any information contained in them, is strictly prohibited and may be illegal. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender directly via email. Any emails that you send to us may be monitored by systems or persons other than the named communicant for the purposes of ascertaining whether the communication complies with the law and company policies.
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.
Re: Determine the backlog in File channel
Posted by Buntu Dev <bu...@gmail.com>.
Awesome thanks!
Whats the right way to calculate the throughput of the channel?
On Mon, Nov 2, 2015 at 4:10 PM, Ahmed Vila <av...@devlogic.eu> wrote:
> To add to it, here is the user manual reference:
> https://flume.apache.org/FlumeUserGuide.html#json-reporting
>
> On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mk...@conversantmedia.com>
> wrote:
>
>> Enable monitoring and you can get the channelFillPercentage and
>> channelSize from the metrics:
>>
>> VM arguments for http monitoring:
>> -Dflume.monitoring.type=http -Dflume.monitoring.port=12345
>>
>> For JMX
>> -Dcom.sun.management.jmxremote.port=12346
>>
>> or whatever port you choose.
>>
>>
>> ------------------------------
>> *From:* Buntu Dev [buntudev@gmail.com]
>> *Sent:* Monday, November 02, 2015 4:33 PM
>> *To:* user@flume.apache.org
>> *Subject:* Determine the backlog in File channel
>>
>> I got a File channel with HDFS sink. In the case when the sink slows down
>> and event taken from the channel falls behind while the event puts continue
>> at the same pace, how would one go about finding the amount of backlog or
>> time it takes to clear the backlog?
>>
>> Thanks!
>>
>> This email and any files included with it may contain privileged,
>> proprietary and/or confidential information that is for the sole use
>> of the intended recipient(s). Any disclosure, copying, distribution,
>> posting, or use of the information contained in or attached to this
>> email is prohibited unless permitted by the sender. If you have
>> received this email in error, please immediately notify the sender
>> via return email, telephone, or fax and destroy this original transmission
>> and its included files without reading or saving it in any manner.
>> Thank you.
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu
> E-mail : avila@devlogic.eu
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
Re: Determine the backlog in File channel
Posted by Ahmed Vila <av...@devlogic.eu>.
To add to it, here is the user manual reference:
https://flume.apache.org/FlumeUserGuide.html#json-reporting
On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mk...@conversantmedia.com>
wrote:
> Enable monitoring and you can get the channelFillPercentage and
> channelSize from the metrics:
>
> VM arguments for http monitoring:
> -Dflume.monitoring.type=http -Dflume.monitoring.port=12345
>
> For JMX
> -Dcom.sun.management.jmxremote.port=12346
>
> or whatever port you choose.
>
>
> ------------------------------
> *From:* Buntu Dev [buntudev@gmail.com]
> *Sent:* Monday, November 02, 2015 4:33 PM
> *To:* user@flume.apache.org
> *Subject:* Determine the backlog in File channel
>
> I got a File channel with HDFS sink. In the case when the sink slows down
> and event taken from the channel falls behind while the event puts continue
> at the same pace, how would one go about finding the amount of backlog or
> time it takes to clear the backlog?
>
> Thanks!
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s). Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender. If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.
>
--
Best regards,
Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina
Office : +387 33 942 123
Mobile: +387 62 139 348
Website: www.devlogic.eu
E-mail : avila@devlogic.eu
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.
--
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.
RE: Determine the backlog in File channel
Posted by "Keane, Mike" <mk...@conversantmedia.com>.
Enable monitoring and you can get the channelFillPercentage and channelSize from the metrics:
VM arguments for http monitoring:
-Dflume.monitoring.type=http -Dflume.monitoring.port=12345
For JMX
-Dcom.sun.management.jmxremote.port=12346
or whatever port you choose.
________________________________
From: Buntu Dev [buntudev@gmail.com]
Sent: Monday, November 02, 2015 4:33 PM
To: user@flume.apache.org
Subject: Determine the backlog in File channel
I got a File channel with HDFS sink. In the case when the sink slows down and event taken from the channel falls behind while the event puts continue at the same pace, how would one go about finding the amount of backlog or time it takes to clear the backlog?
Thanks!
This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s). Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender. If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.