You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2015/04/01 00:00:07 UTC

Re: Sinks without timeout and ELB Spillover - hangs

Hi,

Unfortunately don't have a patch, though I think this is just 1 method that
needs to be added.  Don't think ELB provides the info needed, so one needs
to just have timeout.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> I am not very familiar with ELB, but if you do have a patch that will make
> Flume work better - I’ll be glad to review it. I think the HDFS, HBase,
> Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but
> different impl).
>
>
>
>
> Thanks, Hari
>
> On Tue, Mar 31, 2015 at 9:27 AM, Ashish <pa...@gmail.com> wrote:
>
> > Does ELB provide any kind of status code when it rejects the request
> > or it just doesn't send any response?
> > Perhaps adding a JIRA would be great. Doesn't ELB times out connection
> > which are Idle for more than 60 sec?
> > thanks
> > ashish
> > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic
> > <ot...@gmail.com> wrote:
> >> Hi,
> >>
> >> I don't have a very good bug report, but I want to share one problem
> with
> >> Flume we observed the other day.  Specifically, with Elasticsearch Sink
> and
> >> AWS ELB, but others may have this problem, too.
> >>
> >> In short, ELB has this notion of a Spillover where it can reject
> requests
> >> when it can't handle the load.  Unfortunately, it doesn't actually
> signal
> >> this by closing the connection, so Flume's connection remains open, I
> >> believe in some java.net....... readSocket(....) method.
> >>
> >> To work around this we had to add a socket timeout in our custom
> version of
> >> Flume's Elasticsearch Sink.
> >>
> >> I think it's worth fixing this in Flume and checking if other Sinks need
> >> this sort of timeout.
> >>
> >> HTH
> >>
> >> Otis
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> > --
> > thanks
> > ashish
> > Blog: http://www.ashishpaliwal.com/blog
> > My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Re: Sinks without timeout and ELB Spillover - hangs

Posted by Ashish <pa...@gmail.com>.
Before the thread fades in-memory, here is the JIRA
https://issues.apache.org/jira/browse/FLUME-2661

On Thu, Apr 2, 2015 at 6:26 AM, Ashish <pa...@gmail.com> wrote:
> Is it possible to share the method and class name? It would be
> somewhere in client package I guess.
> I shall do the rest, else would need to re-invent the wheel :)
>
> On Wed, Apr 1, 2015 at 3:30 AM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
>> Hi,
>>
>> Unfortunately don't have a patch, though I think this is just 1 method that
>> needs to be added.  Don't think ELB provides the info needed, so one needs
>> to just have timeout.
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan <hshreedharan@cloudera.com
>>> wrote:
>>
>>> I am not very familiar with ELB, but if you do have a patch that will make
>>> Flume work better - I’ll be glad to review it. I think the HDFS, HBase,
>>> Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but
>>> different impl).
>>>
>>>
>>>
>>>
>>> Thanks, Hari
>>>
>>> On Tue, Mar 31, 2015 at 9:27 AM, Ashish <pa...@gmail.com> wrote:
>>>
>>> > Does ELB provide any kind of status code when it rejects the request
>>> > or it just doesn't send any response?
>>> > Perhaps adding a JIRA would be great. Doesn't ELB times out connection
>>> > which are Idle for more than 60 sec?
>>> > thanks
>>> > ashish
>>> > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic
>>> > <ot...@gmail.com> wrote:
>>> >> Hi,
>>> >>
>>> >> I don't have a very good bug report, but I want to share one problem
>>> with
>>> >> Flume we observed the other day.  Specifically, with Elasticsearch Sink
>>> and
>>> >> AWS ELB, but others may have this problem, too.
>>> >>
>>> >> In short, ELB has this notion of a Spillover where it can reject
>>> requests
>>> >> when it can't handle the load.  Unfortunately, it doesn't actually
>>> signal
>>> >> this by closing the connection, so Flume's connection remains open, I
>>> >> believe in some java.net....... readSocket(....) method.
>>> >>
>>> >> To work around this we had to add a socket timeout in our custom
>>> version of
>>> >> Flume's Elasticsearch Sink.
>>> >>
>>> >> I think it's worth fixing this in Flume and checking if other Sinks need
>>> >> this sort of timeout.
>>> >>
>>> >> HTH
>>> >>
>>> >> Otis
>>> >> --
>>> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> >> Solr & Elasticsearch Support * http://sematext.com/
>>> > --
>>> > thanks
>>> > ashish
>>> > Blog: http://www.ashishpaliwal.com/blog
>>> > My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: Sinks without timeout and ELB Spillover - hangs

Posted by Ashish <pa...@gmail.com>.
Is it possible to share the method and class name? It would be
somewhere in client package I guess.
I shall do the rest, else would need to re-invent the wheel :)

On Wed, Apr 1, 2015 at 3:30 AM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi,
>
> Unfortunately don't have a patch, though I think this is just 1 method that
> needs to be added.  Don't think ELB provides the info needed, so one needs
> to just have timeout.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan <hshreedharan@cloudera.com
>> wrote:
>
>> I am not very familiar with ELB, but if you do have a patch that will make
>> Flume work better - I’ll be glad to review it. I think the HDFS, HBase,
>> Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but
>> different impl).
>>
>>
>>
>>
>> Thanks, Hari
>>
>> On Tue, Mar 31, 2015 at 9:27 AM, Ashish <pa...@gmail.com> wrote:
>>
>> > Does ELB provide any kind of status code when it rejects the request
>> > or it just doesn't send any response?
>> > Perhaps adding a JIRA would be great. Doesn't ELB times out connection
>> > which are Idle for more than 60 sec?
>> > thanks
>> > ashish
>> > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic
>> > <ot...@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> I don't have a very good bug report, but I want to share one problem
>> with
>> >> Flume we observed the other day.  Specifically, with Elasticsearch Sink
>> and
>> >> AWS ELB, but others may have this problem, too.
>> >>
>> >> In short, ELB has this notion of a Spillover where it can reject
>> requests
>> >> when it can't handle the load.  Unfortunately, it doesn't actually
>> signal
>> >> this by closing the connection, so Flume's connection remains open, I
>> >> believe in some java.net....... readSocket(....) method.
>> >>
>> >> To work around this we had to add a socket timeout in our custom
>> version of
>> >> Flume's Elasticsearch Sink.
>> >>
>> >> I think it's worth fixing this in Flume and checking if other Sinks need
>> >> this sort of timeout.
>> >>
>> >> HTH
>> >>
>> >> Otis
>> >> --
>> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> >> Solr & Elasticsearch Support * http://sematext.com/
>> > --
>> > thanks
>> > ashish
>> > Blog: http://www.ashishpaliwal.com/blog
>> > My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal