You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Guillermo Ortiz <ko...@gmail.com> on 2014/09/10 17:52:05 UTC
Load Balancing.
Hi,
I want to use the load balancer in Flume. After reading Flume documentation
about loading balancing I think that it's not really load balacing, if I
understood well, when a sink fail, it chooses a new sink with ronud-robin
or in a random way. But it doesn't distributed on this way all the time
among the differents sink.
Is there any possiblity of distributed the load among all the sinks?
Because if I undertood well, the load balancing option doesn't do it.
Re: Load Balancing.
Posted by Ashish <pa...@gmail.com>.
Source code is best bet. A little more detail is part of the book
http://shop.oreilly.com/product/0636920030348.do, see chapter 6, Sink
Groups and Sink Processor section
On Fri, Sep 12, 2014 at 12:07 PM, Guillermo Ortiz <ko...@gmail.com>
wrote:
> Any documentation or more details of how works??
>
> 2014-09-11 13:53 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
>
>> I guess that there're many open files open in the same time. What means
>> that each event it's delivered some sink randomly.
>>
>> 2014-09-11 10:27 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
>>
>>> I checked, you're right, is there more documentation about it somewhere?
>>> How does it work? I mean, if I choose random, it choose randomly a sink
>>> and when it finishes to generate a file and choose another sink or it send
>>> each event for each sink and really there are many sink generating files
>>> simultaneously?
>>>
>>>
>>> 2014-09-11 3:08 GMT+02:00 Ashish <pa...@gmail.com>:
>>>
>>>> IMHO,
>>>> http://flume.apache.org/FlumeUserGuide.html#load-balancing-sink-processor
>>>> explains it well. Failover Sink processor is used in case of failure event,
>>>> but Load balancing Sink processor would be used to choose the Sink for
>>>> sending Events.
>>>>
>>>> Can try this with a simple topology with two File Roll sinks,
>>>> configured to use Load Balancing Sink Processor.
>>>>
>>>> On Wed, Sep 10, 2014 at 9:22 PM, Guillermo Ortiz <ko...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to use the load balancer in Flume. After reading Flume
>>>>> documentation about loading balancing I think that it's not really load
>>>>> balacing, if I understood well, when a sink fail, it chooses a new sink
>>>>> with ronud-robin or in a random way. But it doesn't distributed on this way
>>>>> all the time among the differents sink.
>>>>>
>>>>> Is there any possiblity of distributed the load among all the sinks?
>>>>> Because if I undertood well, the load balancing option doesn't do it.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> thanks
>>>> ashish
>>>>
>>>> Blog: http://www.ashishpaliwal.com/blog
>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>
>>>
>>>
>>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Load Balancing.
Posted by Guillermo Ortiz <ko...@gmail.com>.
Any documentation or more details of how works??
2014-09-11 13:53 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
> I guess that there're many open files open in the same time. What means
> that each event it's delivered some sink randomly.
>
> 2014-09-11 10:27 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
>
>> I checked, you're right, is there more documentation about it somewhere?
>> How does it work? I mean, if I choose random, it choose randomly a sink
>> and when it finishes to generate a file and choose another sink or it send
>> each event for each sink and really there are many sink generating files
>> simultaneously?
>>
>>
>> 2014-09-11 3:08 GMT+02:00 Ashish <pa...@gmail.com>:
>>
>>> IMHO,
>>> http://flume.apache.org/FlumeUserGuide.html#load-balancing-sink-processor
>>> explains it well. Failover Sink processor is used in case of failure event,
>>> but Load balancing Sink processor would be used to choose the Sink for
>>> sending Events.
>>>
>>> Can try this with a simple topology with two File Roll sinks, configured
>>> to use Load Balancing Sink Processor.
>>>
>>> On Wed, Sep 10, 2014 at 9:22 PM, Guillermo Ortiz <ko...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I want to use the load balancer in Flume. After reading Flume
>>>> documentation about loading balancing I think that it's not really load
>>>> balacing, if I understood well, when a sink fail, it chooses a new sink
>>>> with ronud-robin or in a random way. But it doesn't distributed on this way
>>>> all the time among the differents sink.
>>>>
>>>> Is there any possiblity of distributed the load among all the sinks?
>>>> Because if I undertood well, the load balancing option doesn't do it.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> thanks
>>> ashish
>>>
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>
>>
>>
>
Re: Load Balancing.
Posted by Guillermo Ortiz <ko...@gmail.com>.
I guess that there're many open files open in the same time. What means
that each event it's delivered some sink randomly.
2014-09-11 10:27 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
> I checked, you're right, is there more documentation about it somewhere?
> How does it work? I mean, if I choose random, it choose randomly a sink
> and when it finishes to generate a file and choose another sink or it send
> each event for each sink and really there are many sink generating files
> simultaneously?
>
>
> 2014-09-11 3:08 GMT+02:00 Ashish <pa...@gmail.com>:
>
>> IMHO,
>> http://flume.apache.org/FlumeUserGuide.html#load-balancing-sink-processor
>> explains it well. Failover Sink processor is used in case of failure event,
>> but Load balancing Sink processor would be used to choose the Sink for
>> sending Events.
>>
>> Can try this with a simple topology with two File Roll sinks, configured
>> to use Load Balancing Sink Processor.
>>
>> On Wed, Sep 10, 2014 at 9:22 PM, Guillermo Ortiz <ko...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I want to use the load balancer in Flume. After reading Flume
>>> documentation about loading balancing I think that it's not really load
>>> balacing, if I understood well, when a sink fail, it chooses a new sink
>>> with ronud-robin or in a random way. But it doesn't distributed on this way
>>> all the time among the differents sink.
>>>
>>> Is there any possiblity of distributed the load among all the sinks?
>>> Because if I undertood well, the load balancing option doesn't do it.
>>>
>>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
Re: Load Balancing.
Posted by Guillermo Ortiz <ko...@gmail.com>.
I checked, you're right, is there more documentation about it somewhere?
How does it work? I mean, if I choose random, it choose randomly a sink and
when it finishes to generate a file and choose another sink or it send each
event for each sink and really there are many sink generating files
simultaneously?
2014-09-11 3:08 GMT+02:00 Ashish <pa...@gmail.com>:
> IMHO,
> http://flume.apache.org/FlumeUserGuide.html#load-balancing-sink-processor
> explains it well. Failover Sink processor is used in case of failure event,
> but Load balancing Sink processor would be used to choose the Sink for
> sending Events.
>
> Can try this with a simple topology with two File Roll sinks, configured
> to use Load Balancing Sink Processor.
>
> On Wed, Sep 10, 2014 at 9:22 PM, Guillermo Ortiz <ko...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I want to use the load balancer in Flume. After reading Flume
>> documentation about loading balancing I think that it's not really load
>> balacing, if I understood well, when a sink fail, it chooses a new sink
>> with ronud-robin or in a random way. But it doesn't distributed on this way
>> all the time among the differents sink.
>>
>> Is there any possiblity of distributed the load among all the sinks?
>> Because if I undertood well, the load balancing option doesn't do it.
>>
>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
Re: Load Balancing.
Posted by Ashish <pa...@gmail.com>.
IMHO,
http://flume.apache.org/FlumeUserGuide.html#load-balancing-sink-processor
explains it well. Failover Sink processor is used in case of failure event,
but Load balancing Sink processor would be used to choose the Sink for
sending Events.
Can try this with a simple topology with two File Roll sinks, configured to
use Load Balancing Sink Processor.
On Wed, Sep 10, 2014 at 9:22 PM, Guillermo Ortiz <ko...@gmail.com>
wrote:
> Hi,
>
> I want to use the load balancer in Flume. After reading Flume
> documentation about loading balancing I think that it's not really load
> balacing, if I understood well, when a sink fail, it chooses a new sink
> with ronud-robin or in a random way. But it doesn't distributed on this way
> all the time among the differents sink.
>
> Is there any possiblity of distributed the load among all the sinks?
> Because if I undertood well, the load balancing option doesn't do it.
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal