Handling channel full exception

[mahendran m <ma...@hotmail.com>]

Hi ,
I have started the flume agent with spooling directory source and HDFS sink. how can i know whether connection to HDFS is succeed or not . i have configured log4j.properties to print only error message . when connection to HDFS failed log4j through this exception as WARN so i cannot get this exception when set log4j threshold as ERROR.at some point it lead to channel full exception 
How can i find whether connection to HDFS is succeed or not  ?
Thanks.  		 	   		  

Re: Handling channel full exception

[Ashish <pa...@gmail.com>]

Not sure if I can answer this to your satisfaction, but let me try.
The reason why it logs as warning is, the HDFS can go down even while
the Agent is running and Flume shall keep on retrying to establish the
connection. So if you configuration is correct and HDFS is running, it
would establish the connection.

Second part, even if the Agent is running with connection established
you can still get channel full exception. This would occur if the
event push rate to channel is higher than event drain rate by HDFS
sink.

What do you plan to do by finding the connection state?

Hope it help!

On Mon, Jan 19, 2015 at 10:44 AM, mahendran m <ma...@hotmail.com> wrote:
> Hi ,
>
> I have started the flume agent with spooling directory source and HDFS sink.
> how can i know whether connection to HDFS is succeed or not . i have
> configured log4j.properties to print only error message . when connection to
> HDFS failed log4j through this exception as WARN so i cannot get this
> exception when set log4j threshold as ERROR.at some point it lead to channel
> full exception
>
> How can i find whether connection to HDFS is succeed or not  ?
>
> Thanks.
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal