You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by lancexxx <fe...@gmail.com> on 2012/10/23 09:58:25 UTC
flume-ng use mongdb as sink
hey
I used flume-ng(1.3.0) to collecting webserver log, memory as channel, hdfs as sink, and workd good.
But this time i come up against a series of problems because that i want use mongodb as a final storage , however i can not find any ideas
what use mongodb or other nosql as a sink via google and official guide except a plugin named mongo-hadoop,anyone can help me? thanks very much !
--
lancexxx
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
RE: flume-ng use mongdb as sink
Posted by "Camp, Roy" <rc...@ebay.com>.
Here is a good sink for Flume-NG & Cassandra.
https://github.com/btoddb/flume-ng-cassandra-sink
From: lancexxx [mailto:fervent.linux@gmail.com]
Sent: Tuesday, October 23, 2012 12:58 AM
To: user@flume.apache.org
Subject: flume-ng use mongdb as sink
hey
I used flume-ng(1.3.0) to collecting webserver log, memory as channel, hdfs as sink, and workd good.
But this time i come up against a series of problems because that i want use mongodb as a final storage , however i can not find any ideas
what use mongodb or other nosql as a sink via google and official guide except a plugin named mongo-hadoop,anyone can help me? thanks very much !
--
lancexxx
Sent with Sparrow<http://www.sparrowmailapp.com/?sig>
Re: flume-ng use mongdb as sink
Posted by Alexander Lorenz <wg...@gmail.com>.
Yes, would work with FlumeNG. You've to change in the example the source into avro, and give them a try.
Cheers,
Alex
On Oct 23, 2012, at 4:20 PM, lancexxx <fe...@gmail.com> wrote:
> hi，I found a solution that use mongodb on githua,this is the url:
> https://github.com/leonlee/flume-ng-mongodb-sink
>
> what do you think of it with the avro sink?
> thanks!
> --
> lancexxx
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On 2012年10月23日Tuesday at 下午4:40, Alexander Lorenz wrote:
>
>> Hi,
>>
>> we haven't a mongo sink, so you have to write your own. But you could use a avro sink and import the contents into mongo via http://datasyndrome.com/post/27075009640/from-avro-to-mongo-in-one-line-with-avro-to-mongo-pig?
>>
>> cheers,
>> Alex
>>
>> On Oct 23, 2012, at 9:58 AM, lancexxx <fervent.linux@gmail.com (mailto:fervent.linux@gmail.com)> wrote:
>>
>>>
>>> hey
>>>
>>> I used flume-ng(1.3.0) to collecting webserver log, memory as channel, hdfs as sink, and workd good.
>>> But this time i come up against a series of problems because that i want use mongodb as a final storage , however i can not find any ideas
>>> what use mongodb or other nosql as a sink via google and official guide except a plugin named mongo-hadoop,anyone can help me? thanks very much !
>>>
>>> --
>>> lancexxx
>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>
>>
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: flume-ng use mongdb as sink
Posted by lancexxx <fe...@gmail.com>.
hi，I found a solution that use mongodb on githua,this is the url:
https://github.com/leonlee/flume-ng-mongodb-sink
what do you think of it with the avro sink?
thanks!
--
lancexxx
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On 2012年10月23日Tuesday at 下午4:40, Alexander Lorenz wrote:
> Hi,
>
> we haven't a mongo sink, so you have to write your own. But you could use a avro sink and import the contents into mongo via http://datasyndrome.com/post/27075009640/from-avro-to-mongo-in-one-line-with-avro-to-mongo-pig?
>
> cheers,
> Alex
>
> On Oct 23, 2012, at 9:58 AM, lancexxx <fervent.linux@gmail.com (mailto:fervent.linux@gmail.com)> wrote:
>
> >
> > hey
> >
> > I used flume-ng(1.3.0) to collecting webserver log, memory as channel, hdfs as sink, and workd good.
> > But this time i come up against a series of problems because that i want use mongodb as a final storage , however i can not find any ideas
> > what use mongodb or other nosql as a sink via google and official guide except a plugin named mongo-hadoop,anyone can help me? thanks very much !
> >
> > --
> > lancexxx
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
Re: flume-ng use mongdb as sink
Posted by Alexander Lorenz <wg...@gmail.com>.
Hi,
we haven't a mongo sink, so you have to write your own. But you could use a avro sink and import the contents into mongo via http://datasyndrome.com/post/27075009640/from-avro-to-mongo-in-one-line-with-avro-to-mongo-pig?
cheers,
Alex
On Oct 23, 2012, at 9:58 AM, lancexxx <fe...@gmail.com> wrote:
>
> hey
>
> I used flume-ng(1.3.0) to collecting webserver log, memory as channel, hdfs as sink, and workd good.
> But this time i come up against a series of problems because that i want use mongodb as a final storage , however i can not find any ideas
> what use mongodb or other nosql as a sink via google and official guide except a plugin named mongo-hadoop,anyone can help me? thanks very much !
>
> --
> lancexxx
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF