You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Babu, Prashanth" <Pr...@nttdata.com> on 2012/09/06 23:49:08 UTC
Flume -- InMemory options?
Hi,
Is it possible to use Flume to write to HBase thru InMemory?
I have a steady ingested data of 40,000 events / second.
My requirement is to write to HBase thru Flume and all this in memory.
Basically I am aware that we can write to a log file which will be tailed by Flume and the data is dumped into HBase.
In our requirement, instead of this, we want to directly write to HBase [and if not possible, at least to HDFS].
I had a look at MemoryChannel, but it seems its very limited.
Might not suffice our requirement.
Can someone please shed some light on this?
Did anyone try something like this before?
TIA.
My Environment:
Hadoop - 0.20.203
Flume - 0.90.4
HBase - 0.9.4
Regards,
Prashanth Babu.
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
RE: Flume -- InMemory options?
Posted by "Babu, Prashanth" <Pr...@nttdata.com>.
Thanks Brock. I will have a look at this.
Unfortunately we are on Flume 0.9.4. If we can't achieve this in 0.9.4, then I will check if there is any possibility to move to 1.x.
-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com]
Sent: Friday, September 07, 2012 4:50 AM
To: user@flume.apache.org
Subject: Re: Flume -- InMemory options?
Hi,
I am not very familiar with flume 0.9, but I am with 1.x. It sounds like you won't want to write to a file?
In Flume 1.x you can do this using say AvroSource -> MemoryChannel -> HBase Sink (or AsyncHBase sink).
You'd then use the Flume client or log4j appender to send events to the AvroSource.
Brock
On Thu, Sep 6, 2012 at 4:49 PM, Babu, Prashanth <Pr...@nttdata.com> wrote:
> Hi,
>
> Is it possible to use Flume to write to HBase thru InMemory?
>
> I have a steady ingested data of 40,000 events / second.
> My requirement is to write to HBase thru Flume and all this in memory.
> Basically I am aware that we can write to a log file which will be tailed by Flume and the data is dumped into HBase.
> In our requirement, instead of this, we want to directly write to HBase [and if not possible, at least to HDFS].
>
> I had a look at MemoryChannel, but it seems its very limited.
> Might not suffice our requirement.
>
> Can someone please shed some light on this?
> Did anyone try something like this before?
>
> TIA.
>
> My Environment:
> Hadoop - 0.20.203
> Flume - 0.90.4
> HBase - 0.9.4
>
> Regards,
> Prashanth Babu.
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest
> confidence for the sole use of the addressee and may contain legally
> privileged, confidential, and proprietary data. If you are not the
> intended recipient, please advise the sender by replying promptly to
> this email and then delete and destroy this email and any attachments
> without any further use, copying or forwarding
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Re: Flume -- InMemory options?
Posted by Brock Noland <br...@cloudera.com>.
Hi,
I am not very familiar with flume 0.9, but I am with 1.x. It sounds
like you won't want to write to a file?
In Flume 1.x you can do this using say AvroSource -> MemoryChannel ->
HBase Sink (or AsyncHBase sink).
You'd then use the Flume client or log4j appender to send events to
the AvroSource.
Brock
On Thu, Sep 6, 2012 at 4:49 PM, Babu, Prashanth
<Pr...@nttdata.com> wrote:
> Hi,
>
> Is it possible to use Flume to write to HBase thru InMemory?
>
> I have a steady ingested data of 40,000 events / second.
> My requirement is to write to HBase thru Flume and all this in memory.
> Basically I am aware that we can write to a log file which will be tailed by Flume and the data is dumped into HBase.
> In our requirement, instead of this, we want to directly write to HBase [and if not possible, at least to HDFS].
>
> I had a look at MemoryChannel, but it seems its very limited.
> Might not suffice our requirement.
>
> Can someone please shed some light on this?
> Did anyone try something like this before?
>
> TIA.
>
> My Environment:
> Hadoop - 0.20.203
> Flume - 0.90.4
> HBase - 0.9.4
>
> Regards,
> Prashanth Babu.
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/