You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rahul Ravindran <ra...@yahoo.com> on 2012/11/14 02:52:53 UTC
Flume hops behind HAProxy
Hi,
Before I try it, I wanted to check if there were any known issues with this. We will have multiple flume agents sending an Avro stream each to a smaller set of intermediate flume hops. Are there any issues/concerns around having the flume agents send their streams to an HAProxy which will round robin between the different flume hops. Any issue around the transaction mechanism with this setup?
I know that there is a selector mechanism in Flume to do this, but our operations extensively use HAProxy, and are most familiar with it.
Thanks,
~Rahul.
Re: Flume hops behind HAProxy
Posted by Brock Noland <br...@cloudera.com>.
That will probably work. From my earlier mail:
"I assume it would be connection based round robin? Might work just
fine, but probably best to the use built-in support."
On Thu, Nov 15, 2012 at 11:27 AM, Rahul Ravindran <ra...@yahoo.com> wrote:
> HAProxy has a TCP mode where it round robins TCP connections. Does it need
> to understand something specific about the wire protocol used by Flume?
>
> ________________________________
> From: Brock Noland <br...@cloudera.com>
> To: user@flume.apache.org; Rahul Ravindran <ra...@yahoo.com>
> Sent: Wednesday, November 14, 2012 6:20 PM
>
> Subject: Re: Flume hops behind HAProxy
>
>> It would be round robin but not sticky sessions( so each request could
>> Goto any random flume hop)
>
> Does HAProxy understand the protocol? To round robin requests like
> this it needs to understand the communication protocol, which I
> suppose would work if you were using the HTTPSource.
>
> On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
>> HAProxy is widely deployed already in our deployment and Ops is familiar
>> with dealing with it for hosts which go down etc.
>>
>> ________________________________
>> From: "Camp, Roy" <rc...@ebay.com>
>> To: "user@flume.apache.org" <us...@flume.apache.org>
>> Sent: Wednesday, November 14, 2012 2:15 PM
>> Subject: RE: Flume hops behind HAProxy
>>
>> Out of curiosity, what is the use case vs using the built in load
>> balancing?
>>
>>
>>
>> -----Original Message-----
>> From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
>> Sent: Wednesday, November 14, 2012 1:49 PM
>> To: user@flume.apache.org
>> Cc: user@flume.apache.org
>> Subject: Re: Flume hops behind HAProxy
>>
>> It would be round robin but not sticky sessions( so each request could
>> Goto
>> any random flume hop)
>>
>> Sent from my phone.Excuse the terseness.
>>
>> On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> I assume it would be connection based round robin? Might work just
>>> fine, but probably best to the use built-in support.
>>>
>>> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com>
>>> wrote:
>>>> Resending given I sent it during off-hours.
>>>>
>>>> ________________________________
>>>> From: Rahul Ravindran <ra...@yahoo.com>
>>>> To: "user@flume.apache.org" <us...@flume.apache.org>
>>>> Sent: Tuesday, November 13, 2012 5:52 PM
>>>> Subject: Flume hops behind HAProxy
>>>>
>>>> Hi,
>>>> Before I try it, I wanted to check if there were any known issues
>>>> with this. We will have multiple flume agents sending an Avro stream
>>>> each to a smaller set of intermediate flume hops. Are there any
>>>> issues/concerns around having the flume agents send their streams to
>>>> an HAProxy which will round robin between the different flume hops.
>>>> Any issue around the transaction mechanism with this setup?
>>>>
>>>> I know that there is a selector mechanism in Flume to do this, but
>>>> our operations extensively use HAProxy, and are most familiar with it.
>>>>
>>>> Thanks,
>>>> ~Rahul.
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Rahul Ravindran <ra...@yahoo.com>.
HAProxy has a TCP mode where it round robins TCP connections. Does it need to understand something specific about the wire protocol used by Flume?
________________________________
From: Brock Noland <br...@cloudera.com>
To: user@flume.apache.org; Rahul Ravindran <ra...@yahoo.com>
Sent: Wednesday, November 14, 2012 6:20 PM
Subject: Re: Flume hops behind HAProxy
> It would be round robin but not sticky sessions( so each request could Goto any random flume hop)
Does HAProxy understand the protocol? To round robin requests like
this it needs to understand the communication protocol, which I
suppose would work if you were using the HTTPSource.
On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
> HAProxy is widely deployed already in our deployment and Ops is familiar
> with dealing with it for hosts which go down etc.
>
> ________________________________
> From: "Camp, Roy" <rc...@ebay.com>
> To: "user@flume.apache.org" <us...@flume.apache.org>
> Sent: Wednesday, November 14, 2012 2:15 PM
> Subject: RE: Flume hops behind HAProxy
>
> Out of curiosity, what is the use case vs using the built in load balancing?
>
>
>
> -----Original Message-----
> From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
> Sent: Wednesday, November 14, 2012 1:49 PM
> To: user@flume.apache.org
> Cc: user@flume.apache.org
> Subject: Re: Flume hops behind HAProxy
>
> It would be round robin but not sticky sessions( so each request could Goto
> any random flume hop)
>
> Sent from my phone.Excuse the terseness.
>
> On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> I assume it would be connection based round robin? Might work just
>> fine, but probably best to the use built-in support.
>>
>> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com>
>> wrote:
>>> Resending given I sent it during off-hours.
>>>
>>> ________________________________
>>> From: Rahul Ravindran <ra...@yahoo.com>
>>> To: "user@flume.apache.org" <us...@flume.apache.org>
>>> Sent: Tuesday, November 13, 2012 5:52 PM
>>> Subject: Flume hops behind HAProxy
>>>
>>> Hi,
>>> Before I try it, I wanted to check if there were any known issues
>>> with this. We will have multiple flume agents sending an Avro stream
>>> each to a smaller set of intermediate flume hops. Are there any
>>> issues/concerns around having the flume agents send their streams to
>>> an HAProxy which will round robin between the different flume hops.
>>> Any issue around the transaction mechanism with this setup?
>>>
>>> I know that there is a selector mechanism in Flume to do this, but
>>> our operations extensively use HAProxy, and are most familiar with it.
>>>
>>> Thanks,
>>> ~Rahul.
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Brock Noland <br...@cloudera.com>.
> It would be round robin but not sticky sessions( so each request could Goto any random flume hop)
Does HAProxy understand the protocol? To round robin requests like
this it needs to understand the communication protocol, which I
suppose would work if you were using the HTTPSource.
On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
> HAProxy is widely deployed already in our deployment and Ops is familiar
> with dealing with it for hosts which go down etc.
>
> ________________________________
> From: "Camp, Roy" <rc...@ebay.com>
> To: "user@flume.apache.org" <us...@flume.apache.org>
> Sent: Wednesday, November 14, 2012 2:15 PM
> Subject: RE: Flume hops behind HAProxy
>
> Out of curiosity, what is the use case vs using the built in load balancing?
>
>
>
> -----Original Message-----
> From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
> Sent: Wednesday, November 14, 2012 1:49 PM
> To: user@flume.apache.org
> Cc: user@flume.apache.org
> Subject: Re: Flume hops behind HAProxy
>
> It would be round robin but not sticky sessions( so each request could Goto
> any random flume hop)
>
> Sent from my phone.Excuse the terseness.
>
> On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> I assume it would be connection based round robin? Might work just
>> fine, but probably best to the use built-in support.
>>
>> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com>
>> wrote:
>>> Resending given I sent it during off-hours.
>>>
>>> ________________________________
>>> From: Rahul Ravindran <ra...@yahoo.com>
>>> To: "user@flume.apache.org" <us...@flume.apache.org>
>>> Sent: Tuesday, November 13, 2012 5:52 PM
>>> Subject: Flume hops behind HAProxy
>>>
>>> Hi,
>>> Before I try it, I wanted to check if there were any known issues
>>> with this. We will have multiple flume agents sending an Avro stream
>>> each to a smaller set of intermediate flume hops. Are there any
>>> issues/concerns around having the flume agents send their streams to
>>> an HAProxy which will round robin between the different flume hops.
>>> Any issue around the transaction mechanism with this setup?
>>>
>>> I know that there is a selector mechanism in Flume to do this, but
>>> our operations extensively use HAProxy, and are most familiar with it.
>>>
>>> Thanks,
>>> ~Rahul.
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Rahul Ravindran <ra...@yahoo.com>.
HAProxy is widely deployed already in our deployment and Ops is familiar with dealing with it for hosts which go down etc.
________________________________
From: "Camp, Roy" <rc...@ebay.com>
To: "user@flume.apache.org" <us...@flume.apache.org>
Sent: Wednesday, November 14, 2012 2:15 PM
Subject: RE: Flume hops behind HAProxy
Out of curiosity, what is the use case vs using the built in load balancing?
-----Original Message-----
From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
Sent: Wednesday, November 14, 2012 1:49 PM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: Flume hops behind HAProxy
It would be round robin but not sticky sessions( so each request could Goto any random flume hop)
Sent from my phone.Excuse the terseness.
On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
> I assume it would be connection based round robin? Might work just
> fine, but probably best to the use built-in support.
>
> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
>> Resending given I sent it during off-hours.
>>
>> ________________________________
>> From: Rahul Ravindran <ra...@yahoo.com>
>> To: "user@flume.apache.org" <us...@flume.apache.org>
>> Sent: Tuesday, November 13, 2012 5:52 PM
>> Subject: Flume hops behind HAProxy
>>
>> Hi,
>> Before I try it, I wanted to check if there were any known issues
>> with this. We will have multiple flume agents sending an Avro stream
>> each to a smaller set of intermediate flume hops. Are there any
>> issues/concerns around having the flume agents send their streams to
>> an HAProxy which will round robin between the different flume hops.
>> Any issue around the transaction mechanism with this setup?
>>
>> I know that there is a selector mechanism in Flume to do this, but
>> our operations extensively use HAProxy, and are most familiar with it.
>>
>> Thanks,
>> ~Rahul.
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
RE: Flume hops behind HAProxy
Posted by "Camp, Roy" <rc...@ebay.com>.
Out of curiosity, what is the use case vs using the built in load balancing?
-----Original Message-----
From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
Sent: Wednesday, November 14, 2012 1:49 PM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: Flume hops behind HAProxy
It would be round robin but not sticky sessions( so each request could Goto any random flume hop)
Sent from my phone.Excuse the terseness.
On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
> I assume it would be connection based round robin? Might work just
> fine, but probably best to the use built-in support.
>
> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
>> Resending given I sent it during off-hours.
>>
>> ________________________________
>> From: Rahul Ravindran <ra...@yahoo.com>
>> To: "user@flume.apache.org" <us...@flume.apache.org>
>> Sent: Tuesday, November 13, 2012 5:52 PM
>> Subject: Flume hops behind HAProxy
>>
>> Hi,
>> Before I try it, I wanted to check if there were any known issues
>> with this. We will have multiple flume agents sending an Avro stream
>> each to a smaller set of intermediate flume hops. Are there any
>> issues/concerns around having the flume agents send their streams to
>> an HAProxy which will round robin between the different flume hops.
>> Any issue around the transaction mechanism with this setup?
>>
>> I know that there is a selector mechanism in Flume to do this, but
>> our operations extensively use HAProxy, and are most familiar with it.
>>
>> Thanks,
>> ~Rahul.
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Rahul Ravindran <ra...@yahoo.com>.
It would be round robin but not sticky sessions( so each request could Goto any random flume hop)
Sent from my phone.Excuse the terseness.
On Nov 14, 2012, at 1:33 PM, Brock Noland <br...@cloudera.com> wrote:
> I assume it would be connection based round robin? Might work just
> fine, but probably best to the use built-in support.
>
> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
>> Resending given I sent it during off-hours.
>>
>> ________________________________
>> From: Rahul Ravindran <ra...@yahoo.com>
>> To: "user@flume.apache.org" <us...@flume.apache.org>
>> Sent: Tuesday, November 13, 2012 5:52 PM
>> Subject: Flume hops behind HAProxy
>>
>> Hi,
>> Before I try it, I wanted to check if there were any known issues with
>> this. We will have multiple flume agents sending an Avro stream each to a
>> smaller set of intermediate flume hops. Are there any issues/concerns around
>> having the flume agents send their streams to an HAProxy which will round
>> robin between the different flume hops. Any issue around the transaction
>> mechanism with this setup?
>>
>> I know that there is a selector mechanism in Flume to do this, but our
>> operations extensively use HAProxy, and are most familiar with it.
>>
>> Thanks,
>> ~Rahul.
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Brock Noland <br...@cloudera.com>.
I assume it would be connection based round robin? Might work just
fine, but probably best to the use built-in support.
On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
> Resending given I sent it during off-hours.
>
> ________________________________
> From: Rahul Ravindran <ra...@yahoo.com>
> To: "user@flume.apache.org" <us...@flume.apache.org>
> Sent: Tuesday, November 13, 2012 5:52 PM
> Subject: Flume hops behind HAProxy
>
> Hi,
> Before I try it, I wanted to check if there were any known issues with
> this. We will have multiple flume agents sending an Avro stream each to a
> smaller set of intermediate flume hops. Are there any issues/concerns around
> having the flume agents send their streams to an HAProxy which will round
> robin between the different flume hops. Any issue around the transaction
> mechanism with this setup?
>
> I know that there is a selector mechanism in Flume to do this, but our
> operations extensively use HAProxy, and are most familiar with it.
>
> Thanks,
> ~Rahul.
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume hops behind HAProxy
Posted by Rahul Ravindran <ra...@yahoo.com>.
Resending given I sent it during off-hours.
________________________________
From: Rahul Ravindran <ra...@yahoo.com>
To: "user@flume.apache.org" <us...@flume.apache.org>
Sent: Tuesday, November 13, 2012 5:52 PM
Subject: Flume hops behind HAProxy
Hi,
Before I try it, I wanted to check if there were any known issues with this. We will have multiple flume agents sending an Avro stream each to a smaller set of intermediate flume hops. Are there any issues/concerns around having the flume agents send their streams to an HAProxy which will round robin between the different flume hops. Any issue around the transaction mechanism with this setup?
I know that there is a selector mechanism in Flume to do this, but our operations extensively use HAProxy, and are most familiar with it.
Thanks,
~Rahul.