Flume error handling

[Sverre Bakke <sv...@gmail.com>]

Hi,

When creating a new EventDrivenSource running as an executor, what is
the correct approach to handling shutdown gracefully?

I am writing a custom source that will poll a compressed file line by
line using BufferedReader and pushing these lines to a
ChannelProcessor using processEvent(). This as a result of Spooling
Directory not supporting compressed files. This also means that most
of the time, my Flume source will be blocking on
BufferedReader.readLine() or blocking on
ChannelProcessor.processEvent().

If I shutdown the executor from the stop() method of my source, the
typical response from Flume will be that the ChannelProcessor will
generate a ChannelException. In what situations can I expect that the
ChannelException actually is the result of a shutdown (e.g. ctrl+c)
rather than some other issue that should be handled as a truly
exceptional situation/error? Or am I approaching graceful shutdown
completely wrong?

Is there any specific order in which the Flume sources, interceptors
and sinks are signaled to shut down?

I feel that when it comes to error handling (and shutdowns), the
developer guide and javadoc is a bit lacking unfortunately.

Regards,
Sverre Bakke

Re: Flume error handling

[Jeff Lord <jl...@cloudera.com>]

Hi Sverre,

Have you taken a look at the EventDrivenSourceRunner ?
I think this *may* help.

https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/EventDrivenSourceRunner.java

-Jeff

On Thu, Nov 6, 2014 at 4:44 AM, Sverre Bakke <sv...@gmail.com> wrote:

> Hi,
>
> When creating a new EventDrivenSource running as an executor, what is
> the correct approach to handling shutdown gracefully?
>
> I am writing a custom source that will poll a compressed file line by
> line using BufferedReader and pushing these lines to a
> ChannelProcessor using processEvent(). This as a result of Spooling
> Directory not supporting compressed files. This also means that most
> of the time, my Flume source will be blocking on
> BufferedReader.readLine() or blocking on
> ChannelProcessor.processEvent().
>
> If I shutdown the executor from the stop() method of my source, the
> typical response from Flume will be that the ChannelProcessor will
> generate a ChannelException. In what situations can I expect that the
> ChannelException actually is the result of a shutdown (e.g. ctrl+c)
> rather than some other issue that should be handled as a truly
> exceptional situation/error? Or am I approaching graceful shutdown
> completely wrong?
>
> Is there any specific order in which the Flume sources, interceptors
> and sinks are signaled to shut down?
>
> I feel that when it comes to error handling (and shutdowns), the
> developer guide and javadoc is a bit lacking unfortunately.
>
> Regards,
> Sverre Bakke
>