Flume restart after sync destination failure

[saverio mucci <sa...@gmail.com>]

Hi all!
I'm a new user to the ML and to Flume too.

I started using flume to make sure that all my data gets into hbase from
the frontend http server and i'm quite satisfied.
The problem i got is connected to the moments when hbase does not respond
correctly like a crash/or network problem then flume keep storing the
events but does not send them to the synk without any error in the log file
just increasing the size of the queue.
All starts to work correctly again only when i restart the flume agent that
then starts to flush the enqueued data.

I'm using flume 1.4.0

Thank you all in advance
Best
Saverio Mucci

Re: Flume restart after sync destination failure

[Sharninder <sh...@gmail.com>]

slightly unrelated but I've seen the flume file source also "hang" a few
times on me. Restarting solved my problem. Can't really say more since I
don't have a lot of data but flume failing on you is a problem and is the
reason a lot of people develop their own "custom collectors".

On a related note, what are my options for monitoring data flow in a flume
pipeline ?




On Fri, Jun 13, 2014 at 5:09 PM, saverio mucci <sa...@gmail.com>
wrote:

> Yep and it keeps queuing objects until u do not restarts the agent then it
> does resume the normal  behaviour.
>
> The problem is that is giving no errors during this period like it does
> neither try to send the data :/
> Il 13/giu/2014 05:10 "Roshan Naik" <ro...@hortonworks.com> ha scritto:
>
> Are you saying that flume does not resume normal operation after a HBase
>> connectivity is restored ?
>> -roshan
>>
>>
>> On Thu, Jun 12, 2014 at 3:29 AM, saverio mucci <sa...@gmail.com>
>> wrote:
>>
>>> Hi all!
>>> I'm a new user to the ML and to Flume too.
>>>
>>> I started using flume to make sure that all my data gets into hbase from
>>> the frontend http server and i'm quite satisfied.
>>> The problem i got is connected to the moments when hbase does not
>>> respond correctly like a crash/or network problem then flume keep storing
>>> the events but does not send them to the synk without any error in the log
>>> file just increasing the size of the queue.
>>> All starts to work correctly again only when i restart the flume agent
>>> that then starts to flush the enqueued data.
>>>
>>> I'm using flume 1.4.0
>>>
>>> Thank you all in advance
>>> Best
>>> Saverio Mucci
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>

Re: Flume restart after sync destination failure

[saverio mucci <sa...@gmail.com>]

Yep and it keeps queuing objects until u do not restarts the agent then it
does resume the normal  behaviour.

The problem is that is giving no errors during this period like it does
neither try to send the data :/
Il 13/giu/2014 05:10 "Roshan Naik" <ro...@hortonworks.com> ha scritto:

> Are you saying that flume does not resume normal operation after a HBase
> connectivity is restored ?
> -roshan
>
>
> On Thu, Jun 12, 2014 at 3:29 AM, saverio mucci <sa...@gmail.com>
> wrote:
>
>> Hi all!
>> I'm a new user to the ML and to Flume too.
>>
>> I started using flume to make sure that all my data gets into hbase from
>> the frontend http server and i'm quite satisfied.
>> The problem i got is connected to the moments when hbase does not respond
>> correctly like a crash/or network problem then flume keep storing the
>> events but does not send them to the synk without any error in the log file
>> just increasing the size of the queue.
>> All starts to work correctly again only when i restart the flume agent
>> that then starts to flush the enqueued data.
>>
>> I'm using flume 1.4.0
>>
>> Thank you all in advance
>> Best
>> Saverio Mucci
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Flume restart after sync destination failure

[Roshan Naik <ro...@hortonworks.com>]

Are you saying that flume does not resume normal operation after a HBase
connectivity is restored ?
-roshan


On Thu, Jun 12, 2014 at 3:29 AM, saverio mucci <sa...@gmail.com>
wrote:

> Hi all!
> I'm a new user to the ML and to Flume too.
>
> I started using flume to make sure that all my data gets into hbase from
> the frontend http server and i'm quite satisfied.
> The problem i got is connected to the moments when hbase does not respond
> correctly like a crash/or network problem then flume keep storing the
> events but does not send them to the synk without any error in the log file
> just increasing the size of the queue.
> All starts to work correctly again only when i restart the flume agent
> that then starts to flush the enqueued data.
>
> I'm using flume 1.4.0
>
> Thank you all in advance
> Best
> Saverio Mucci
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.