Flume idea

[Rami Mankevich <ra...@gmail.com>]

Hey Guys.
I m working iwth flume for couple of months and faces several issues:
My flow is simple:
Im using REST WS which has internal AVRO client.
So the flow is simple :
HTTP webservce -> AVRO CLIENT ------------> Fume with AVRO source
The problem starts when i started using File Channel Since now all the HTTP
requests to the REST WS are blocked on file system .
With memory channel evrything was fast and perfect,
So i started thinking about solution and the idea was of using memcached
between all the Tier 1 flume servers.
This way i will not be blocked on file system from one side and from other
side i will still be hightly avalible.
Does it make sence?

In general i implemented teh solution....
Just wanted to know you opinion.

Re: Flume idea

[Roshan Naik <ro...@hortonworks.com>]

Could you elaborate on "using memcached between all the Tier 1 flume
servers."
It is not clear where memcached is fitting in the data flow pipeline.
-roshan


On Thu, Sep 26, 2013 at 2:08 AM, Rami Mankevich <ra...@gmail.com>wrote:

> Hey Guys.
> I m working iwth flume for couple of months and faces several issues:
> My flow is simple:
> Im using REST WS which has internal AVRO client.
> So the flow is simple :
> HTTP webservce -> AVRO CLIENT ------------> Fume with AVRO source
> The problem starts when i started using File Channel Since now all the HTTP
> requests to the REST WS are blocked on file system .
> With memory channel evrything was fast and perfect,
> So i started thinking about solution and the idea was of using memcached
> between all the Tier 1 flume servers.
> This way i will not be blocked on file system from one side and from other
> side i will still be hightly avalible.
> Does it make sence?
>
> In general i implemented teh solution....
> Just wanted to know you opinion.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.