You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Ajay Srivastava <Aj...@guavus.com> on 2016/01/18 10:21:02 UTC

Installing flume in distributed mode and HA

Hi,

I am new to flume and am looking for the documentation to configure flume in distributed mode. There was no section present for this in user guide.
I came across this page on cloudera http://archive.cloudera.com/cdh/3/flume/UserGuide/ which describes steps to configure flume in fully distributed mode. Since I could not find these instructions on apache website, I am wondering if CDH has its own version of flume. Do these configuration steps work well for apache flume as well ?



Regards,
Ajay

Re: Installing flume in distributed mode and HA

Posted by Gonzalo Herreros <gh...@gmail.com>.

I mean for instance if you use an http source then have a HW or SW load
balancer so it can call any of the agents.
Yes, you have to worry about HA at source, channel and sink. Depending on
the use case, it might be enough to have it at the source.

On 18 January 2016 at 09:39, Ajay Srivastava <Aj...@guavus.com>
wrote:

> Thanks Gonzalo for quick reply.
>
> By load balancer, do you mean load balancing group of flume agents ?
> If yes, I do need to take care of HA for sources, channels and sinks too.
> Am I correct ?
>
>
>
> Regards,
> Ajay
>
>
> On 18-Jan-2016, at 2:58 pm, Gonzalo Herreros <gh...@gmail.com> wrote:
>
> That Cloudera documentation is ancient and talks about the old Flume
> (that's why it differs from what you see in the Apache website), the modern
> Flume (also called Flume-ng) doesn't not have a master, to have HA you need
> several agents with the same configuration and a load balancer in front.
> In some cases specific cases you might need need to build a hierarchy and
> even add Kafka channels you have true fail tolerances.
>
> Regards,
> Gonzalo
>
> On 18 January 2016 at 09:21, Ajay Srivastava <Aj...@guavus.com>
> wrote:
>
>> Hi,
>>
>> I am new to flume and am looking for the documentation to configure flume
>> in distributed mode. There was no section present for this in user guide.
>> I came across this page on cloudera
>> http://archive.cloudera.com/cdh/3/flume/UserGuide/ which describes steps
>> to configure flume in fully distributed mode. Since I could not find these
>> instructions on apache website, I am wondering if CDH has its own version
>> of flume. Do these configuration steps work well for apache flume as well ?
>>
>>
>>
>> Regards,
>> Ajay
>>
>
>
>

Re: Installing flume in distributed mode and HA

Posted by Ajay Srivastava <Aj...@guavus.com>.

Thanks Gonzalo for quick reply.

By load balancer, do you mean load balancing group of flume agents ?
If yes, I do need to take care of HA for sources, channels and sinks too. Am I correct ?



Regards,
Ajay


On 18-Jan-2016, at 2:58 pm, Gonzalo Herreros <gh...@gmail.com>> wrote:

That Cloudera documentation is ancient and talks about the old Flume (that's why it differs from what you see in the Apache website), the modern Flume (also called Flume-ng) doesn't not have a master, to have HA you need several agents with the same configuration and a load balancer in front.
In some cases specific cases you might need need to build a hierarchy and even add Kafka channels you have true fail tolerances.

Regards,
Gonzalo

On 18 January 2016 at 09:21, Ajay Srivastava <Aj...@guavus.com>> wrote:
Hi,

I am new to flume and am looking for the documentation to configure flume in distributed mode. There was no section present for this in user guide.
I came across this page on cloudera http://archive.cloudera.com/cdh/3/flume/UserGuide/ which describes steps to configure flume in fully distributed mode. Since I could not find these instructions on apache website, I am wondering if CDH has its own version of flume. Do these configuration steps work well for apache flume as well ?



Regards,
Ajay

Re: Installing flume in distributed mode and HA

Posted by Gonzalo Herreros <gh...@gmail.com>.

That Cloudera documentation is ancient and talks about the old Flume
(that's why it differs from what you see in the Apache website), the modern
Flume (also called Flume-ng) doesn't not have a master, to have HA you need
several agents with the same configuration and a load balancer in front.
In some cases specific cases you might need need to build a hierarchy and
even add Kafka channels you have true fail tolerances.

Regards,
Gonzalo

On 18 January 2016 at 09:21, Ajay Srivastava <Aj...@guavus.com>
wrote:

> Hi,
>
> I am new to flume and am looking for the documentation to configure flume
> in distributed mode. There was no section present for this in user guide.
> I came across this page on cloudera
> http://archive.cloudera.com/cdh/3/flume/UserGuide/ which describes steps
> to configure flume in fully distributed mode. Since I could not find these
> instructions on apache website, I am wondering if CDH has its own version
> of flume. Do these configuration steps work well for apache flume as well ?
>
>
>
> Regards,
> Ajay
>