Is it possible to check if there are events in a channel?

[Andrei <fa...@gmail.com>]

Are there any tools to monitor state of a Flume channel and, specifically,
determine the moment when there are no more events?

To give you some context: I have a batch of events that I want to pass
through Flume and eventually get to HDFS via corresponding sink. When all
events of the batch are dumped, I want to perform post-processing routine
on the entire batch. My assumption is that if channel is empty for a while,
all events of the batch are dumped and I can start the routing. But how can
I determine if channel is empty?

Re: Is it possible to check if there are events in a channel?

[Andrei <fa...@gmail.com>]

Thanks, it looks interesting. I'll play around with it a bit and see if I
can get information I need.


On Tue, Aug 27, 2013 at 5:33 PM, Israel Ekpo <is...@aicer.org> wrote:

> Andrei,
>
> Check out the monitoring tool here and see if it helps
>
> http://flume.apache.org/FlumeUserGuide.html#monitoring
>
>
> *Author and Instructor for the Upcoming Book and Lecture Series*
> *Massive Log Data Aggregation, Processing, Searching and Visualization
> with Open Source Software*
> *http://massivelogdata.com*
>
>
> On 27 August 2013 06:22, Andrei <fa...@gmail.com> wrote:
>
>> Hi Juhani,
>>
>> thanks for your answer. Can you please tell me how can I use JMX to get
>> fill percentage? I have very poor knowledge of JMX, but as far as I
>> understand it helps to monitor specific resources used by JVM. So what
>> resource should I monitor to to get fill percentage?
>>
>>
>> On Tue, Aug 27, 2013 at 11:24 AM, Juhani Connolly <
>> juhani_connolly@cyberagent.co.jp> wrote:
>>
>>> You can get the channel capacity and fill percentage using jmx. You'd
>>> need to regularly poll the fill percentage.
>>>
>>>
>>> On 08/26/2013 08:41 PM, Andrei wrote:
>>>
>>>> Are there any tools to monitor state of a Flume channel and,
>>>> specifically, determine the moment when there are no more events?
>>>>
>>>> To give you some context: I have a batch of events that I want to pass
>>>> through Flume and eventually get to HDFS via corresponding sink. When all
>>>> events of the batch are dumped, I want to perform post-processing routine
>>>> on the entire batch. My assumption is that if channel is empty for a while,
>>>> all events of the batch are dumped and I can start the routing. But how can
>>>> I determine if channel is empty?
>>>>
>>>
>>>
>>
>

Re: Is it possible to check if there are events in a channel?

[Israel Ekpo <is...@aicer.org>]

Andrei,

Check out the monitoring tool here and see if it helps

http://flume.apache.org/FlumeUserGuide.html#monitoring


*Author and Instructor for the Upcoming Book and Lecture Series*
*Massive Log Data Aggregation, Processing, Searching and Visualization with
Open Source Software*
*http://massivelogdata.com*


On 27 August 2013 06:22, Andrei <fa...@gmail.com> wrote:

> Hi Juhani,
>
> thanks for your answer. Can you please tell me how can I use JMX to get
> fill percentage? I have very poor knowledge of JMX, but as far as I
> understand it helps to monitor specific resources used by JVM. So what
> resource should I monitor to to get fill percentage?
>
>
> On Tue, Aug 27, 2013 at 11:24 AM, Juhani Connolly <
> juhani_connolly@cyberagent.co.jp> wrote:
>
>> You can get the channel capacity and fill percentage using jmx. You'd
>> need to regularly poll the fill percentage.
>>
>>
>> On 08/26/2013 08:41 PM, Andrei wrote:
>>
>>> Are there any tools to monitor state of a Flume channel and,
>>> specifically, determine the moment when there are no more events?
>>>
>>> To give you some context: I have a batch of events that I want to pass
>>> through Flume and eventually get to HDFS via corresponding sink. When all
>>> events of the batch are dumped, I want to perform post-processing routine
>>> on the entire batch. My assumption is that if channel is empty for a while,
>>> all events of the batch are dumped and I can start the routing. But how can
>>> I determine if channel is empty?
>>>
>>
>>
>

Re: Is it possible to check if there are events in a channel?

[Andrei <fa...@gmail.com>]

Hi Juhani,

thanks for your answer. Can you please tell me how can I use JMX to get
fill percentage? I have very poor knowledge of JMX, but as far as I
understand it helps to monitor specific resources used by JVM. So what
resource should I monitor to to get fill percentage?


On Tue, Aug 27, 2013 at 11:24 AM, Juhani Connolly <
juhani_connolly@cyberagent.co.jp> wrote:

> You can get the channel capacity and fill percentage using jmx. You'd need
> to regularly poll the fill percentage.
>
>
> On 08/26/2013 08:41 PM, Andrei wrote:
>
>> Are there any tools to monitor state of a Flume channel and,
>> specifically, determine the moment when there are no more events?
>>
>> To give you some context: I have a batch of events that I want to pass
>> through Flume and eventually get to HDFS via corresponding sink. When all
>> events of the batch are dumped, I want to perform post-processing routine
>> on the entire batch. My assumption is that if channel is empty for a while,
>> all events of the batch are dumped and I can start the routing. But how can
>> I determine if channel is empty?
>>
>
>

Re: Is it possible to check if there are events in a channel?

[Juhani Connolly <ju...@cyberagent.co.jp>]

You can get the channel capacity and fill percentage using jmx. You'd 
need to regularly poll the fill percentage.

On 08/26/2013 08:41 PM, Andrei wrote:
> Are there any tools to monitor state of a Flume channel and, 
> specifically, determine the moment when there are no more events?
>
> To give you some context: I have a batch of events that I want to pass 
> through Flume and eventually get to HDFS via corresponding sink. When 
> all events of the batch are dumped, I want to perform post-processing 
> routine on the entire batch. My assumption is that if channel is empty 
> for a while, all events of the batch are dumped and I can start the 
> routing. But how can I determine if channel is empty?