You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Sri Ramya <ra...@gmail.com> on 2013/01/23 18:59:17 UTC
flume-cassandra
hi
How to intigrate flume with cassandra by using logsandra plugin.
thank you.
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
hi
I am unable to talk with cassandra using flume-ng sink. Can you tell me
what is the configuration file to give to flume agent.
On Thu, Jan 31, 2013 at 4:26 PM, Sri Ramya <ra...@gmail.com> wrote:
> thanks for ur support,
> i will get back to u for farther help
>
>
> On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its
>> own Cluster of hosts is correct. Just as all the other Flume Sinks, the
>> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to
>> Cassandra using one of the supported protocols. I believe we are using
>> Hector but you could use Astyanax, Thrift or I suppose even CQL. You could
>> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor
>> it to meet your needs.
>>
>> Ralph
>>
>>
>> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>>
>>
>> I am running flume-ng on one host and Cassandra on another host. How i
>> have to connect it. Which host i have to install flume-casandra sink? Do i
>> need to run flume-ng on the host which is having Cassandra?
>>
>> Please reply me...
>>
>>
>> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <ralph.goers@dslextreme.com
>> > wrote:
>>
>>> We ended up writing our own custom sink for Cassandra so that we could
>>> customize the fields, write to Cassandra indexes and multiple column
>>> families. We primarily use Cassandra to store log records and we insure
>>> they can't be duplicated by generating a guid in the Appender that writes
>>> to Flume (via Avro). The guid is then used as the row key.
>>>
>>> What do you mean by separation?
>>>
>>> We use Log4j 2's Flume Appender with the RFC5424Layout and
>>> StructuredDataMessages (see EventLogger). All the individual items in the
>>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
>>> the FlumeEvent data map. The Cassandra Sink then writes these to
>>> individual columns in Cassandra. The Cassandra Sink also writes these
>>> items to the appropriate indexes so that we can perform whatever queries
>>> that perform well. The ipAddress is definitely one of the items we are
>>> interested in and capture in our indexes.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>>>
>>> hi Ralph
>>>
>>> I am able to integrate flume with Cassandra. But the problem is ' if i
>>> send a single event to Cassandra to store, it is storing multiple row in
>>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>>> i am staring work with Flume-ng.
>>>
>>> Make me clear in one thing, Is there any separation of storing data in
>>> Cassandra based on flume-ng agent(IP address)???
>>>
>>> Thank you
>>>
>>>
>>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <
>>> ralph.goers@dslextreme.com> wrote:
>>>
>>>> This is absolutely NOT correct. We are using Flume NG to log to
>>>> Cassandra. We fought with Flume OG for months trying to get it to work
>>>> but it was never stable. The architecture of Flume OG had serious
>>>> problems, which is why Flume NG was created.
>>>>
>>>> Ralph
>>>>
>>>>
>>>>
>>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>>
>>>> Hi ,
>>>> Can you give details like your input format and all ...
>>>> I think going with flume 0.9.4 for cassandra is better.
>>>>
>>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>
>>>>> What do you mean by "collector"?
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>
>>>>>> Thank you very much. But I need a collector in my application,
>>>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>>>> flum. Now i am getting a problem with integrating flume collector with
>>>>>> cassandra. If any body tried it before please help me.
>>>>>> thank in advance.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>>>
>>>>>>> Hi Sri,
>>>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mike
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Thank you very much.
>>>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>>>> flume-ng . And which is the better option to use.
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You can use develop your own customized sink which can do
>>>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Som
>>>>>>>>>
>>>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com>wrote:
>>>>>>>>>
>>>>>>>>>> You may also want to check out
>>>>>>>>>>
>>>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>>>
>>>>>>>>>> Roy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>>>> To: user@flume.apache.org
>>>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>>>
>>>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration? How did that go?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>>>> > hi
>>>>>>>>>> >
>>>>>>>>>> > How to intigrate flume with cassandra by using logsandra
>>>>>>>>>> plugin.
>>>>>>>>>> >
>>>>>>>>>> > thank you.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
thanks for ur support,
i will get back to u for farther help
On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers <ra...@dslextreme.com>wrote:
> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its
> own Cluster of hosts is correct. Just as all the other Flume Sinks, the
> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to
> Cassandra using one of the supported protocols. I believe we are using
> Hector but you could use Astyanax, Thrift or I suppose even CQL. You could
> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor
> it to meet your needs.
>
> Ralph
>
>
> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>
>
> I am running flume-ng on one host and Cassandra on another host. How i
> have to connect it. Which host i have to install flume-casandra sink? Do i
> need to run flume-ng on the host which is having Cassandra?
>
> Please reply me...
>
>
> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> We ended up writing our own custom sink for Cassandra so that we could
>> customize the fields, write to Cassandra indexes and multiple column
>> families. We primarily use Cassandra to store log records and we insure
>> they can't be duplicated by generating a guid in the Appender that writes
>> to Flume (via Avro). The guid is then used as the row key.
>>
>> What do you mean by separation?
>>
>> We use Log4j 2's Flume Appender with the RFC5424Layout and
>> StructuredDataMessages (see EventLogger). All the individual items in the
>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
>> the FlumeEvent data map. The Cassandra Sink then writes these to
>> individual columns in Cassandra. The Cassandra Sink also writes these
>> items to the appropriate indexes so that we can perform whatever queries
>> that perform well. The ipAddress is definitely one of the items we are
>> interested in and capture in our indexes.
>>
>> Ralph
>>
>>
>>
>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>>
>> hi Ralph
>>
>> I am able to integrate flume with Cassandra. But the problem is ' if i
>> send a single event to Cassandra to store, it is storing multiple row in
>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>> i am staring work with Flume-ng.
>>
>> Make me clear in one thing, Is there any separation of storing data in
>> Cassandra based on flume-ng agent(IP address)???
>>
>> Thank you
>>
>>
>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ralph.goers@dslextreme.com
>> > wrote:
>>
>>> This is absolutely NOT correct. We are using Flume NG to log to
>>> Cassandra. We fought with Flume OG for months trying to get it to work
>>> but it was never stable. The architecture of Flume OG had serious
>>> problems, which is why Flume NG was created.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>
>>> Hi ,
>>> Can you give details like your input format and all ...
>>> I think going with flume 0.9.4 for cassandra is better.
>>>
>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>
>>>> What do you mean by "collector"?
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>
>>>>> Thank you very much. But I need a collector in my application,
>>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>>> flum. Now i am getting a problem with integrating flume collector with
>>>>> cassandra. If any body tried it before please help me.
>>>>> thank in advance.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>>
>>>>>> Hi Sri,
>>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>>
>>>>>> Regards,
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>>
>>>>>>> Thank you very much.
>>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>>> flume-ng . And which is the better option to use.
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>>
>>>>>>>> You can use develop your own customized sink which can do
>>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Som
>>>>>>>>
>>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com>wrote:
>>>>>>>>
>>>>>>>>> You may also want to check out
>>>>>>>>>
>>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>>
>>>>>>>>> Roy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>>> To: user@flume.apache.org
>>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>>
>>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration? How did that go?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>>> > hi
>>>>>>>>> >
>>>>>>>>> > How to intigrate flume with cassandra by using logsandra
>>>>>>>>> plugin.
>>>>>>>>> >
>>>>>>>>> > thank you.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: flume-cassandra
Posted by Ralph Goers <ra...@dslextreme.com>.
Running Flume-ng on one host (or cluster of hosts) and Cassandra on its own Cluster of hosts is correct. Just as all the other Flume Sinks, the Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to Cassandra using one of the supported protocols. I believe we are using Hector but you could use Astyanax, Thrift or I suppose even CQL. You could start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor it to meet your needs.
Ralph
On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote:
>
> I am running flume-ng on one host and Cassandra on another host. How i have to connect it. Which host i have to install flume-casandra sink? Do i need to run flume-ng on the host which is having Cassandra?
>
> Please reply me...
>
>
> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <ra...@dslextreme.com> wrote:
> We ended up writing our own custom sink for Cassandra so that we could customize the fields, write to Cassandra indexes and multiple column families. We primarily use Cassandra to store log records and we insure they can't be duplicated by generating a guid in the Appender that writes to Flume (via Avro). The guid is then used as the row key.
>
> What do you mean by separation?
>
> We use Log4j 2's Flume Appender with the RFC5424Layout and StructuredDataMessages (see EventLogger). All the individual items in the StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to the FlumeEvent data map. The Cassandra Sink then writes these to individual columns in Cassandra. The Cassandra Sink also writes these items to the appropriate indexes so that we can perform whatever queries that perform well. The ipAddress is definitely one of the items we are interested in and capture in our indexes.
>
> Ralph
>
>
>
> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>
>> hi Ralph
>>
>> I am able to integrate flume with Cassandra. But the problem is ' if i send a single event to Cassandra to store, it is storing multiple row in Cassnadra. Thats why i thought of shifting to logsandra. But its very difficult to to use that logsandra plug in. Any way by your suggestionsNow i am staring work with Flume-ng.
>>
>> Make me clear in one thing, Is there any separation of storing data in Cassandra based on flume-ng agent(IP address)???
>>
>> Thank you
>>
>>
>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com> wrote:
>> This is absolutely NOT correct. We are using Flume NG to log to Cassandra. We fought with Flume OG for months trying to get it to work but it was never stable. The architecture of Flume OG had serious problems, which is why Flume NG was created.
>>
>> Ralph
>>
>>
>>
>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>
>>> Hi ,
>>> Can you give details like your input format and all ...
>>> I think going with flume 0.9.4 for cassandra is better.
>>>
>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>>> What do you mean by "collector"?
>>>
>>>
>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>> Thank you very much. But I need a collector in my application, flume-ng does not have any collector. Thats why i started with Cloudeera flum. Now i am getting a problem with integrating flume collector with cassandra. If any body tried it before please help me.
>>> thank in advance.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
>>> Hi Sri,
>>> Cloudera originally created Flume, then contributed it to the Apache Software Foundation (ASF), and continues to invest heavily into it under the auspices of the ASF. The current generation of Flume is called Flume NG. I encourage you to use the latest "NG" generation of Flume (version 1.x) over the so-called "OG" generation (version 0.9.x).
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>>> Thank you very much.
>>> Can u tell me whats the difference between cloudera flume and flume-ng . And which is the better option to use.
>>> Thank you.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com> wrote:
>>> You can use develop your own customized sink which can do pre-processing in flume and then send the events to cassandra
>>>
>>> Regards,
>>> Som
>>>
>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>> You may also want to check out
>>>
>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>
>>> Roy
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: rektide [mailto:rektide@voodoowarez.com]
>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>> To: user@flume.apache.org
>>> Subject: Re: flume-cassandra
>>>
>>> Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
>>>
>>> Thanks.
>>>
>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>> > hi
>>> >
>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>> >
>>> > thank you.
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
I am running flume-ng on one host and Cassandra on another host. How i
have to connect it. Which host i have to install flume-casandra sink? Do i
need to run flume-ng on the host which is having Cassandra?
Please reply me...
On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <ra...@dslextreme.com>wrote:
> We ended up writing our own custom sink for Cassandra so that we could
> customize the fields, write to Cassandra indexes and multiple column
> families. We primarily use Cassandra to store log records and we insure
> they can't be duplicated by generating a guid in the Appender that writes
> to Flume (via Avro). The guid is then used as the row key.
>
> What do you mean by separation?
>
> We use Log4j 2's Flume Appender with the RFC5424Layout and
> StructuredDataMessages (see EventLogger). All the individual items in the
> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
> the FlumeEvent data map. The Cassandra Sink then writes these to
> individual columns in Cassandra. The Cassandra Sink also writes these
> items to the appropriate indexes so that we can perform whatever queries
> that perform well. The ipAddress is definitely one of the items we are
> interested in and capture in our indexes.
>
> Ralph
>
>
>
> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>
> hi Ralph
>
> I am able to integrate flume with Cassandra. But the problem is ' if i
> send a single event to Cassandra to store, it is storing multiple row in
> Cassnadra. Thats why i thought of shifting to logsandra. But its very
> difficult to to use that logsandra plug in. Any way by your suggestionsNow
> i am staring work with Flume-ng.
>
> Make me clear in one thing, Is there any separation of storing data in
> Cassandra based on flume-ng agent(IP address)???
>
> Thank you
>
>
> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> This is absolutely NOT correct. We are using Flume NG to log to
>> Cassandra. We fought with Flume OG for months trying to get it to work
>> but it was never stable. The architecture of Flume OG had serious
>> problems, which is why Flume NG was created.
>>
>> Ralph
>>
>>
>>
>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>
>> Hi ,
>> Can you give details like your input format and all ...
>> I think going with flume 0.9.4 for cassandra is better.
>>
>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> What do you mean by "collector"?
>>>
>>>
>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> Thank you very much. But I need a collector in my application,
>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>> flum. Now i am getting a problem with integrating flume collector with
>>>> cassandra. If any body tried it before please help me.
>>>> thank in advance.
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>
>>>>> Hi Sri,
>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>
>>>>> Regards,
>>>>> Mike
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>
>>>>>> Thank you very much.
>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>> flume-ng . And which is the better option to use.
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>
>>>>>>> You can use develop your own customized sink which can do
>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>
>>>>>>> Regards,
>>>>>>> Som
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>>
>>>>>>>> You may also want to check out
>>>>>>>>
>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>
>>>>>>>> Roy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>> To: user@flume.apache.org
>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>
>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>>> How did that go?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>> > hi
>>>>>>>> >
>>>>>>>> > How to intigrate flume with cassandra by using logsandra
>>>>>>>> plugin.
>>>>>>>> >
>>>>>>>> > thank you.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
Re: flume-cassandra
Posted by Yogi Nerella <yn...@gmail.com>.
Simply superb, appreciate your help for taking time to answer all my
questions.
Thanks again
Yogi
On Tue, Jan 29, 2013 at 1:59 PM, Ralph Goers <ra...@dslextreme.com>wrote:
> See below.
>
> On Jan 29, 2013, at 11:43 AM, Yogi Nerella wrote:
>
> Ralph,
>
> Sorry Ramya for side tracking this thread.
>
> I am also looking for some open source tools to collect all log records
> from various applications into a central place.
> Looking at Cassandra File System as well, but really do not understand
> what value we get by storing in it, instead of storing in NFS based file
> system.
>
>
> Cassandra is not a file system. It is a NoSQL solution that stores the
> data elements as columns in rows. See
> http://nosql.mypopescu.com/post/2981945438/why-netflix-picked-amazon-simpledb-hadoop-hbase-and for
> a nice discussion on the differences between Hadoop, Cassandra and SimpleDB.
>
>
>
> Are there any good UI tools for searching the CFS database for
> administrators to look at errors?
>
>
> No. That is similar to asking if there are good tools to looking for
> "xxxx" in database "yyyy" where "xxxx" is something specific to your
> problem. As a NoSQL database Cassandra is not solving a specific problem
> such as capturing logs.
>
>
> Is MDC working with flume log4jappender? I couldnt make it work?
>
>
> Flume's Log4j appender is for Log4j 1.x. I am working on Log4j 2 which is
> a complete rewrite and is designed for modern JDKs and to be able to
> reliably handle audit logging. See
> http://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAppender.
> Log4j 2 supports both writing to a remote agent over Avro or an embedded
> agent.
>
>
> log4jappender of flume has so many dependencies, are there any good
> "SocketAppenders", which can store and forward in the case of the server
> down?
>
>
> One of the unfortunate aspects of Flume is that it does have a lot of
> dependencies. When you use the Avro remote Appender you will only need a
> few jars from Log4j, Flume, Avro and possibly a couple more that they
> depend on. If you use the embedded appender you will need quite a few more
> since you are essentially running a Flume agent in your application.
>
> Generic socket appenders generally suffer from the problem that they
> cannot guarantee delivery. IOW, you get control back as soon as TCP says
> the data was delivered. It might never make it into the Flume channel.
> When you use Avro you do get guaranteed delivery as you don't get control
> back until the data is written to the Flume channel. A "Socket Appender"
> that can store and forward would be the Flume embedded appender in Log4j 2.
>
>
> Can I use "SocketAppender" and send the data to "flume agent" configured
> to receive netcat messages?
>
>
> Yes, you can use a Socket Appender. Unlike other logging frameworks, with
> Log4j 2 you can use a Layout to format the data going over the socket
> connection in any manner you want.
>
> Ralph
>
Re: flume-cassandra
Posted by Ralph Goers <ra...@dslextreme.com>.
See below.
On Jan 29, 2013, at 11:43 AM, Yogi Nerella wrote:
> Ralph,
>
> Sorry Ramya for side tracking this thread.
>
> I am also looking for some open source tools to collect all log records from various applications into a central place.
> Looking at Cassandra File System as well, but really do not understand what value we get by storing in it, instead of storing in NFS based file system.
Cassandra is not a file system. It is a NoSQL solution that stores the data elements as columns in rows. See http://nosql.mypopescu.com/post/2981945438/why-netflix-picked-amazon-simpledb-hadoop-hbase-and for a nice discussion on the differences between Hadoop, Cassandra and SimpleDB.
>
> Are there any good UI tools for searching the CFS database for administrators to look at errors?
No. That is similar to asking if there are good tools to looking for "xxxx" in database "yyyy" where "xxxx" is something specific to your problem. As a NoSQL database Cassandra is not solving a specific problem such as capturing logs.
>
> Is MDC working with flume log4jappender? I couldnt make it work?
Flume's Log4j appender is for Log4j 1.x. I am working on Log4j 2 which is a complete rewrite and is designed for modern JDKs and to be able to reliably handle audit logging. See http://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAppender. Log4j 2 supports both writing to a remote agent over Avro or an embedded agent.
>
> log4jappender of flume has so many dependencies, are there any good "SocketAppenders", which can store and forward in the case of the server down?
One of the unfortunate aspects of Flume is that it does have a lot of dependencies. When you use the Avro remote Appender you will only need a few jars from Log4j, Flume, Avro and possibly a couple more that they depend on. If you use the embedded appender you will need quite a few more since you are essentially running a Flume agent in your application.
Generic socket appenders generally suffer from the problem that they cannot guarantee delivery. IOW, you get control back as soon as TCP says the data was delivered. It might never make it into the Flume channel. When you use Avro you do get guaranteed delivery as you don't get control back until the data is written to the Flume channel. A "Socket Appender" that can store and forward would be the Flume embedded appender in Log4j 2.
>
> Can I use "SocketAppender" and send the data to "flume agent" configured to receive netcat messages?
Yes, you can use a Socket Appender. Unlike other logging frameworks, with Log4j 2 you can use a Layout to format the data going over the socket connection in any manner you want.
Ralph
Re: flume-cassandra
Posted by Yogi Nerella <yn...@gmail.com>.
Ralph,
Sorry Ramya for side tracking this thread.
I am also looking for some open source tools to collect all log records
from various applications into a central place.
Looking at Cassandra File System as well, but really do not understand what
value we get by storing in it, instead of storing in NFS based file system.
Are there any good UI tools for searching the CFS database for
administrators to look at errors?
Is MDC working with flume log4jappender? I couldnt make it work?
log4jappender of flume has so many dependencies, are there any good
"SocketAppenders", which can store and forward in the case of the server
down?
Can I use "SocketAppender" and send the data to "flume agent" configured to
receive netcat messages?
Yogi
On Tue, Jan 29, 2013 at 11:24 AM, Ralph Goers <ra...@dslextreme.com>wrote:
> We ended up writing our own custom sink for Cassandra so that we could
> customize the fields, write to Cassandra indexes and multiple column
> families. We primarily use Cassandra to store log records and we insure
> they can't be duplicated by generating a guid in the Appender that writes
> to Flume (via Avro). The guid is then used as the row key.
>
> What do you mean by separation?
>
> We use Log4j 2's Flume Appender with the RFC5424Layout and
> StructuredDataMessages (see EventLogger). All the individual items in the
> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to
> the FlumeEvent data map. The Cassandra Sink then writes these to
> individual columns in Cassandra. The Cassandra Sink also writes these
> items to the appropriate indexes so that we can perform whatever queries
> that perform well. The ipAddress is definitely one of the items we are
> interested in and capture in our indexes.
>
> Ralph
>
>
>
> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
>
> hi Ralph
>
> I am able to integrate flume with Cassandra. But the problem is ' if i
> send a single event to Cassandra to store, it is storing multiple row in
> Cassnadra. Thats why i thought of shifting to logsandra. But its very
> difficult to to use that logsandra plug in. Any way by your suggestionsNow
> i am staring work with Flume-ng.
>
> Make me clear in one thing, Is there any separation of storing data in
> Cassandra based on flume-ng agent(IP address)???
>
> Thank you
>
>
> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> This is absolutely NOT correct. We are using Flume NG to log to
>> Cassandra. We fought with Flume OG for months trying to get it to work
>> but it was never stable. The architecture of Flume OG had serious
>> problems, which is why Flume NG was created.
>>
>> Ralph
>>
>>
>>
>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>
>> Hi ,
>> Can you give details like your input format and all ...
>> I think going with flume 0.9.4 for cassandra is better.
>>
>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> What do you mean by "collector"?
>>>
>>>
>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> Thank you very much. But I need a collector in my application,
>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>> flum. Now i am getting a problem with integrating flume collector with
>>>> cassandra. If any body tried it before please help me.
>>>> thank in advance.
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>
>>>>> Hi Sri,
>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>
>>>>> Regards,
>>>>> Mike
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>
>>>>>> Thank you very much.
>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>> flume-ng . And which is the better option to use.
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>
>>>>>>> You can use develop your own customized sink which can do
>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>
>>>>>>> Regards,
>>>>>>> Som
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>>
>>>>>>>> You may also want to check out
>>>>>>>>
>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>
>>>>>>>> Roy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>> To: user@flume.apache.org
>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>
>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>>> How did that go?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>> > hi
>>>>>>>> >
>>>>>>>> > How to intigrate flume with cassandra by using logsandra
>>>>>>>> plugin.
>>>>>>>> >
>>>>>>>> > thank you.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
Re: flume-cassandra
Posted by Ralph Goers <ra...@dslextreme.com>.
We ended up writing our own custom sink for Cassandra so that we could customize the fields, write to Cassandra indexes and multiple column families. We primarily use Cassandra to store log records and we insure they can't be duplicated by generating a guid in the Appender that writes to Flume (via Avro). The guid is then used as the row key.
What do you mean by separation?
We use Log4j 2's Flume Appender with the RFC5424Layout and StructuredDataMessages (see EventLogger). All the individual items in the StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to the FlumeEvent data map. The Cassandra Sink then writes these to individual columns in Cassandra. The Cassandra Sink also writes these items to the appropriate indexes so that we can perform whatever queries that perform well. The ipAddress is definitely one of the items we are interested in and capture in our indexes.
Ralph
On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:
> hi Ralph
>
> I am able to integrate flume with Cassandra. But the problem is ' if i send a single event to Cassandra to store, it is storing multiple row in Cassnadra. Thats why i thought of shifting to logsandra. But its very difficult to to use that logsandra plug in. Any way by your suggestionsNow i am staring work with Flume-ng.
>
> Make me clear in one thing, Is there any separation of storing data in Cassandra based on flume-ng agent(IP address)???
>
> Thank you
>
>
> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com> wrote:
> This is absolutely NOT correct. We are using Flume NG to log to Cassandra. We fought with Flume OG for months trying to get it to work but it was never stable. The architecture of Flume OG had serious problems, which is why Flume NG was created.
>
> Ralph
>
>
>
> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>
>> Hi ,
>> Can you give details like your input format and all ...
>> I think going with flume 0.9.4 for cassandra is better.
>>
>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>> What do you mean by "collector"?
>>
>>
>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>> Thank you very much. But I need a collector in my application, flume-ng does not have any collector. Thats why i started with Cloudeera flum. Now i am getting a problem with integrating flume collector with cassandra. If any body tried it before please help me.
>> thank in advance.
>>
>>
>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
>> Hi Sri,
>> Cloudera originally created Flume, then contributed it to the Apache Software Foundation (ASF), and continues to invest heavily into it under the auspices of the ASF. The current generation of Flume is called Flume NG. I encourage you to use the latest "NG" generation of Flume (version 1.x) over the so-called "OG" generation (version 0.9.x).
>>
>> Regards,
>> Mike
>>
>>
>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>> Thank you very much.
>> Can u tell me whats the difference between cloudera flume and flume-ng . And which is the better option to use.
>> Thank you.
>>
>>
>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com> wrote:
>> You can use develop your own customized sink which can do pre-processing in flume and then send the events to cassandra
>>
>> Regards,
>> Som
>>
>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>> You may also want to check out
>>
>> https://github.com/btoddb/flume-ng-cassandra-sink
>>
>> Roy
>>
>>
>>
>> -----Original Message-----
>> From: rektide [mailto:rektide@voodoowarez.com]
>> Sent: Wednesday, January 23, 2013 10:24 AM
>> To: user@flume.apache.org
>> Subject: Re: flume-cassandra
>>
>> Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
>>
>> Thanks.
>>
>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>> > hi
>> >
>> > How to intigrate flume with cassandra by using logsandra plugin.
>> >
>> > thank you.
>>
>>
>>
>>
>>
>>
>
>
Re: flume-cassandra
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi,
As the error message says, you haven't a valid log4j config. Start your flume-ng agent with the additional option "--conf" and point this into the directory which have a log4j.properties file
-Alex
On Jan 29, 2013, at 6:15 AM, Sri Ramya <ra...@gmail.com> wrote:
> HI
>
> i am unable to configure flume-ng, i am geting a warning
>
> + exec /home/sriramya/project/jdk1.7.0_07/bin/java -Xmx20m
> -Dflume.root.logger=DEBUG,console -cp
> './conf/:/home/sriramya/Desktop/apache-flume-1.3.1-bin/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf
> -n agent1
> log4j:WARN No appenders could be found for logger
> (org.apache.flume.node.Application).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
> and its get terminated. Please tell me how to start
>
>
> On Tue, Jan 29, 2013 at 9:24 AM, Sri Ramya <ra...@gmail.com> wrote:
>
>> hi Ralph
>>
>> I am able to integrate flume with Cassandra. But the problem is ' if i
>> send a single event to Cassandra to store, it is storing multiple row in
>> Cassnadra. Thats why i thought of shifting to logsandra. But its very
>> difficult to to use that logsandra plug in. Any way by your suggestionsNow
>> i am staring work with Flume-ng.
>>
>> Make me clear in one thing, Is there any separation of storing data in
>> Cassandra based on flume-ng agent(IP address)???
>>
>> Thank you
>>
>>
>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com>wrote:
>>
>>> This is absolutely NOT correct. We are using Flume NG to log to
>>> Cassandra. We fought with Flume OG for months trying to get it to work
>>> but it was never stable. The architecture of Flume OG had serious
>>> problems, which is why Flume NG was created.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>>
>>> Hi ,
>>> Can you give details like your input format and all ...
>>> I think going with flume 0.9.4 for cassandra is better.
>>>
>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>>>
>>>> What do you mean by "collector"?
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>>
>>>>> Thank you very much. But I need a collector in my application,
>>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>>> flum. Now i am getting a problem with integrating flume collector with
>>>>> cassandra. If any body tried it before please help me.
>>>>> thank in advance.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>>
>>>>>> Hi Sri,
>>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>>
>>>>>> Regards,
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>>
>>>>>>> Thank you very much.
>>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>>> flume-ng . And which is the better option to use.
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>>
>>>>>>>> You can use develop your own customized sink which can do
>>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Som
>>>>>>>>
>>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>>>
>>>>>>>>> You may also want to check out
>>>>>>>>>
>>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>>
>>>>>>>>> Roy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>>> To: user@flume.apache.org
>>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>>
>>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>>>> How did that go?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>>>> hi
>>>>>>>>>>
>>>>>>>>>> How to intigrate flume with cassandra by using logsandra
>>>>>>>>> plugin.
>>>>>>>>>>
>>>>>>>>>> thank you.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
HI
i am unable to configure flume-ng, i am geting a warning
+ exec /home/sriramya/project/jdk1.7.0_07/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'./conf/:/home/sriramya/Desktop/apache-flume-1.3.1-bin/lib/*'
-Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf
-n agent1
log4j:WARN No appenders could be found for logger
(org.apache.flume.node.Application).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
and its get terminated. Please tell me how to start
On Tue, Jan 29, 2013 at 9:24 AM, Sri Ramya <ra...@gmail.com> wrote:
> hi Ralph
>
> I am able to integrate flume with Cassandra. But the problem is ' if i
> send a single event to Cassandra to store, it is storing multiple row in
> Cassnadra. Thats why i thought of shifting to logsandra. But its very
> difficult to to use that logsandra plug in. Any way by your suggestionsNow
> i am staring work with Flume-ng.
>
> Make me clear in one thing, Is there any separation of storing data in
> Cassandra based on flume-ng agent(IP address)???
>
> Thank you
>
>
> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> This is absolutely NOT correct. We are using Flume NG to log to
>> Cassandra. We fought with Flume OG for months trying to get it to work
>> but it was never stable. The architecture of Flume OG had serious
>> problems, which is why Flume NG was created.
>>
>> Ralph
>>
>>
>>
>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>>
>> Hi ,
>> Can you give details like your input format and all ...
>> I think going with flume 0.9.4 for cassandra is better.
>>
>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> What do you mean by "collector"?
>>>
>>>
>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> Thank you very much. But I need a collector in my application,
>>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>>> flum. Now i am getting a problem with integrating flume collector with
>>>> cassandra. If any body tried it before please help me.
>>>> thank in advance.
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>>
>>>>> Hi Sri,
>>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>>
>>>>> Regards,
>>>>> Mike
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>>
>>>>>> Thank you very much.
>>>>>> Can u tell me whats the difference between cloudera flume and
>>>>>> flume-ng . And which is the better option to use.
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <
>>>>>> shekhar2581@gmail.com> wrote:
>>>>>>
>>>>>>> You can use develop your own customized sink which can do
>>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>>
>>>>>>> Regards,
>>>>>>> Som
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>>
>>>>>>>> You may also want to check out
>>>>>>>>
>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>>
>>>>>>>> Roy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>>> To: user@flume.apache.org
>>>>>>>> Subject: Re: flume-cassandra
>>>>>>>>
>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>>> How did that go?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>>> > hi
>>>>>>>> >
>>>>>>>> > How to intigrate flume with cassandra by using logsandra
>>>>>>>> plugin.
>>>>>>>> >
>>>>>>>> > thank you.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
hi Ralph
I am able to integrate flume with Cassandra. But the problem is ' if i send
a single event to Cassandra to store, it is storing multiple row in
Cassnadra. Thats why i thought of shifting to logsandra. But its very
difficult to to use that logsandra plug in. Any way by your suggestionsNow
i am staring work with Flume-ng.
Make me clear in one thing, Is there any separation of storing data in
Cassandra based on flume-ng agent(IP address)???
Thank you
On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ra...@dslextreme.com>wrote:
> This is absolutely NOT correct. We are using Flume NG to log to
> Cassandra. We fought with Flume OG for months trying to get it to work
> but it was never stable. The architecture of Flume OG had serious
> problems, which is why Flume NG was created.
>
> Ralph
>
>
>
> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
>
> Hi ,
> Can you give details like your input format and all ...
> I think going with flume 0.9.4 for cassandra is better.
>
> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>
>> What do you mean by "collector"?
>>
>>
>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> Thank you very much. But I need a collector in my application,
>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>> flum. Now i am getting a problem with integrating flume collector with
>>> cassandra. If any body tried it before please help me.
>>> thank in advance.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>
>>>> Hi Sri,
>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>
>>>>> Thank you very much.
>>>>> Can u tell me whats the difference between cloudera flume and flume-ng
>>>>> . And which is the better option to use.
>>>>> Thank you.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <shekhar2581@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> You can use develop your own customized sink which can do
>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>
>>>>>> Regards,
>>>>>> Som
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>
>>>>>>> You may also want to check out
>>>>>>>
>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>
>>>>>>> Roy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>> To: user@flume.apache.org
>>>>>>> Subject: Re: flume-cassandra
>>>>>>>
>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>> How did that go?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>> > hi
>>>>>>> >
>>>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>>>> >
>>>>>>> > thank you.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
Re: flume-cassandra
Posted by Ralph Goers <ra...@dslextreme.com>.
This is absolutely NOT correct. We are using Flume NG to log to Cassandra. We fought with Flume OG for months trying to get it to work but it was never stable. The architecture of Flume OG had serious problems, which is why Flume NG was created.
Ralph
On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
> Hi ,
> Can you give details like your input format and all ...
> I think going with flume 0.9.4 for cassandra is better.
>
> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
> What do you mean by "collector"?
>
>
> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
> Thank you very much. But I need a collector in my application, flume-ng does not have any collector. Thats why i started with Cloudeera flum. Now i am getting a problem with integrating flume collector with cassandra. If any body tried it before please help me.
> thank in advance.
>
>
> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
> Hi Sri,
> Cloudera originally created Flume, then contributed it to the Apache Software Foundation (ASF), and continues to invest heavily into it under the auspices of the ASF. The current generation of Flume is called Flume NG. I encourage you to use the latest "NG" generation of Flume (version 1.x) over the so-called "OG" generation (version 0.9.x).
>
> Regards,
> Mike
>
>
> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
> Thank you very much.
> Can u tell me whats the difference between cloudera flume and flume-ng . And which is the better option to use.
> Thank you.
>
>
> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com> wrote:
> You can use develop your own customized sink which can do pre-processing in flume and then send the events to cassandra
>
> Regards,
> Som
>
> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
> You may also want to check out
>
> https://github.com/btoddb/flume-ng-cassandra-sink
>
> Roy
>
>
>
> -----Original Message-----
> From: rektide [mailto:rektide@voodoowarez.com]
> Sent: Wednesday, January 23, 2013 10:24 AM
> To: user@flume.apache.org
> Subject: Re: flume-cassandra
>
> Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
>
> Thanks.
>
> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
> > hi
> >
> > How to intigrate flume with cassandra by using logsandra plugin.
> >
> > thank you.
>
>
>
>
>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
Actually i am able to integrate flume and Cassandra
by using simpleCassndraSink but i want to use logsandra to integrate.
Input formate, i don't have any problem with that.
On Thu, Jan 24, 2013 at 11:16 AM, Priyanka Jain
<ja...@gmail.com>wrote:
> Hi ,
> Can you give details like your input format and all ...
> I think going with flume 0.9.4 for cassandra is better.
>
> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>
>> What do you mean by "collector"?
>>
>>
>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> Thank you very much. But I need a collector in my application,
>>> flume-ng does not have any collector. Thats why i started with Cloudeera
>>> flum. Now i am getting a problem with integrating flume collector with
>>> cassandra. If any body tried it before please help me.
>>> thank in advance.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>
>>>> Hi Sri,
>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>
>>>>> Thank you very much.
>>>>> Can u tell me whats the difference between cloudera flume and flume-ng
>>>>> . And which is the better option to use.
>>>>> Thank you.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <shekhar2581@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> You can use develop your own customized sink which can do
>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>
>>>>>> Regards,
>>>>>> Som
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>
>>>>>>> You may also want to check out
>>>>>>>
>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>
>>>>>>> Roy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>> To: user@flume.apache.org
>>>>>>> Subject: Re: flume-cassandra
>>>>>>>
>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>> How did that go?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>> > hi
>>>>>>> >
>>>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>>>> >
>>>>>>> > thank you.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flume-cassandra
Posted by Priyanka Jain <ja...@gmail.com>.
Hi ,
Can you give details like your input format and all ...
I think going with flume 0.9.4 for cassandra is better.
On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
> What do you mean by "collector"?
>
>
> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>
>> Thank you very much. But I need a collector in my application,
>> flume-ng does not have any collector. Thats why i started with Cloudeera
>> flum. Now i am getting a problem with integrating flume collector with
>> cassandra. If any body tried it before please help me.
>> thank in advance.
>>
>>
>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> Hi Sri,
>>> Cloudera originally created Flume, then contributed it to the Apache
>>> Software Foundation (ASF), and continues to invest heavily into it under
>>> the auspices of the ASF. The current generation of Flume is called Flume
>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> Thank you very much.
>>>> Can u tell me whats the difference between cloudera flume and flume-ng
>>>> . And which is the better option to use.
>>>> Thank you.
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
>>>>
>>>>> You can use develop your own customized sink which can do
>>>>> pre-processing in flume and then send the events to cassandra
>>>>>
>>>>> Regards,
>>>>> Som
>>>>>
>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>
>>>>>> You may also want to check out
>>>>>>
>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>
>>>>>> Roy
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>> To: user@flume.apache.org
>>>>>> Subject: Re: flume-cassandra
>>>>>>
>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>> How did that go?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>> > hi
>>>>>> >
>>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>>> >
>>>>>> > thank you.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flume-cassandra
Posted by Mike Percy <mp...@cloudera.com>.
Hi Sri,
Flume NG has a single role called an agent. Agents can be configured into
arbitrary topologies, such as a fan-in (converging) topology like you
describe.
I think it would help you to read up on the architecture of Flume NG here:
https://blogs.apache.org/flume/entry/flume_ng_architecture - note that this
article is more than a year old but it describes something of why Flume was
re-architected and what the implications are. At this point, Flume NG has
stabilized and it can handle virtually any use case that Flume OG could
handle. Plus, Flume NG is more reliable, more supported (i.e. if you find a
bug, it will get fixed), and easier to keep running. Of course, it is your
call to do as you wish.
Regards,
Mike
On Wed, Jan 23, 2013 at 9:45 PM, Sri Ramya <ra...@gmail.com> wrote:
> flume agents sends data to one host which is called flume collector. Its
> there in cloudera flume.
>
>
> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
>
>> What do you mean by "collector"?
>>
>>
>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> Thank you very much. But I need a collector in my application, flume-ng
>>> does not have any collector. Thats why i started with Cloudeera flum. Now i
>>> am getting a problem with integrating flume collector with cassandra. If
>>> any body tried it before please help me.
>>> thank in advance.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com>wrote:
>>>
>>>> Hi Sri,
>>>> Cloudera originally created Flume, then contributed it to the Apache
>>>> Software Foundation (ASF), and continues to invest heavily into it under
>>>> the auspices of the ASF. The current generation of Flume is called Flume
>>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com>wrote:
>>>>
>>>>> Thank you very much.
>>>>> Can u tell me whats the difference between cloudera flume and flume-ng
>>>>> . And which is the better option to use.
>>>>> Thank you.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <shekhar2581@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> You can use develop your own customized sink which can do
>>>>>> pre-processing in flume and then send the events to cassandra
>>>>>>
>>>>>> Regards,
>>>>>> Som
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>>
>>>>>>> You may also want to check out
>>>>>>>
>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>>
>>>>>>> Roy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>>> To: user@flume.apache.org
>>>>>>> Subject: Re: flume-cassandra
>>>>>>>
>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>>> How did that go?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>>> > hi
>>>>>>> >
>>>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>>>> >
>>>>>>> > thank you.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
flume agents sends data to one host which is called flume collector. Its
there in cloudera flume.
On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mp...@cloudera.com> wrote:
> What do you mean by "collector"?
>
>
> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
>
>> Thank you very much. But I need a collector in my application, flume-ng
>> does not have any collector. Thats why i started with Cloudeera flum. Now i
>> am getting a problem with integrating flume collector with cassandra. If
>> any body tried it before please help me.
>> thank in advance.
>>
>>
>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> Hi Sri,
>>> Cloudera originally created Flume, then contributed it to the Apache
>>> Software Foundation (ASF), and continues to invest heavily into it under
>>> the auspices of the ASF. The current generation of Flume is called Flume
>>> NG. I encourage you to use the latest "NG" generation of Flume (version
>>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> Thank you very much.
>>>> Can u tell me whats the difference between cloudera flume and flume-ng
>>>> . And which is the better option to use.
>>>> Thank you.
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
>>>>
>>>>> You can use develop your own customized sink which can do
>>>>> pre-processing in flume and then send the events to cassandra
>>>>>
>>>>> Regards,
>>>>> Som
>>>>>
>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>>
>>>>>> You may also want to check out
>>>>>>
>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>>
>>>>>> Roy
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>>> To: user@flume.apache.org
>>>>>> Subject: Re: flume-cassandra
>>>>>>
>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>>> How did that go?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>>> > hi
>>>>>> >
>>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>>> >
>>>>>> > thank you.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flume-cassandra
Posted by Mike Percy <mp...@cloudera.com>.
What do you mean by "collector"?
On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ra...@gmail.com> wrote:
> Thank you very much. But I need a collector in my application, flume-ng
> does not have any collector. Thats why i started with Cloudeera flum. Now i
> am getting a problem with integrating flume collector with cassandra. If
> any body tried it before please help me.
> thank in advance.
>
>
> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
>
>> Hi Sri,
>> Cloudera originally created Flume, then contributed it to the Apache
>> Software Foundation (ASF), and continues to invest heavily into it under
>> the auspices of the ASF. The current generation of Flume is called Flume
>> NG. I encourage you to use the latest "NG" generation of Flume (version
>> 1.x) over the so-called "OG" generation (version 0.9.x).
>>
>> Regards,
>> Mike
>>
>>
>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> Thank you very much.
>>> Can u tell me whats the difference between cloudera flume and flume-ng .
>>> And which is the better option to use.
>>> Thank you.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
>>>
>>>> You can use develop your own customized sink which can do
>>>> pre-processing in flume and then send the events to cassandra
>>>>
>>>> Regards,
>>>> Som
>>>>
>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>>
>>>>> You may also want to check out
>>>>>
>>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>>
>>>>> Roy
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>>> To: user@flume.apache.org
>>>>> Subject: Re: flume-cassandra
>>>>>
>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ?
>>>>> How did that go?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>>> > hi
>>>>> >
>>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>>> >
>>>>> > thank you.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
Thank you very much. But I need a collector in my application, flume-ng
does not have any collector. Thats why i started with Cloudeera flum. Now i
am getting a problem with integrating flume collector with cassandra. If
any body tried it before please help me.
thank in advance.
On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mp...@cloudera.com> wrote:
> Hi Sri,
> Cloudera originally created Flume, then contributed it to the Apache
> Software Foundation (ASF), and continues to invest heavily into it under
> the auspices of the ASF. The current generation of Flume is called Flume
> NG. I encourage you to use the latest "NG" generation of Flume (version
> 1.x) over the so-called "OG" generation (version 0.9.x).
>
> Regards,
> Mike
>
>
> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
>
>> Thank you very much.
>> Can u tell me whats the difference between cloudera flume and flume-ng .
>> And which is the better option to use.
>> Thank you.
>>
>>
>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
>>
>>> You can use develop your own customized sink which can do pre-processing
>>> in flume and then send the events to cassandra
>>>
>>> Regards,
>>> Som
>>>
>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>>
>>>> You may also want to check out
>>>>
>>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>>
>>>> Roy
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: rektide [mailto:rektide@voodoowarez.com]
>>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>>> To: user@flume.apache.org
>>>> Subject: Re: flume-cassandra
>>>>
>>>> Have you tried https://github.com/thobbs/logsandra#configuration ? How
>>>> did that go?
>>>>
>>>> Thanks.
>>>>
>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>>> > hi
>>>> >
>>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>>> >
>>>> > thank you.
>>>>
>>>
>>>
>>
>
Re: flume-cassandra
Posted by Mike Percy <mp...@cloudera.com>.
Hi Sri,
Cloudera originally created Flume, then contributed it to the Apache
Software Foundation (ASF), and continues to invest heavily into it under
the auspices of the ASF. The current generation of Flume is called Flume
NG. I encourage you to use the latest "NG" generation of Flume (version
1.x) over the so-called "OG" generation (version 0.9.x).
Regards,
Mike
On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ra...@gmail.com> wrote:
> Thank you very much.
> Can u tell me whats the difference between cloudera flume and flume-ng .
> And which is the better option to use.
> Thank you.
>
>
> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
>
>> You can use develop your own customized sink which can do pre-processing
>> in flume and then send the events to cassandra
>>
>> Regards,
>> Som
>>
>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>>
>>> You may also want to check out
>>>
>>> https://github.com/btoddb/flume-ng-cassandra-sink
>>>
>>> Roy
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: rektide [mailto:rektide@voodoowarez.com]
>>> Sent: Wednesday, January 23, 2013 10:24 AM
>>> To: user@flume.apache.org
>>> Subject: Re: flume-cassandra
>>>
>>> Have you tried https://github.com/thobbs/logsandra#configuration ? How
>>> did that go?
>>>
>>> Thanks.
>>>
>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>>> > hi
>>> >
>>> > How to intigrate flume with cassandra by using logsandra plugin.
>>> >
>>> > thank you.
>>>
>>
>>
>
Re: flume-cassandra
Posted by Sri Ramya <ra...@gmail.com>.
Thank you very much.
Can u tell me whats the difference between cloudera flume and flume-ng .
And which is the better option to use.
Thank you.
On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <sh...@gmail.com>wrote:
> You can use develop your own customized sink which can do pre-processing
> in flume and then send the events to cassandra
>
> Regards,
> Som
>
> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
>
>> You may also want to check out
>>
>> https://github.com/btoddb/flume-ng-cassandra-sink
>>
>> Roy
>>
>>
>>
>> -----Original Message-----
>> From: rektide [mailto:rektide@voodoowarez.com]
>> Sent: Wednesday, January 23, 2013 10:24 AM
>> To: user@flume.apache.org
>> Subject: Re: flume-cassandra
>>
>> Have you tried https://github.com/thobbs/logsandra#configuration ? How
>> did that go?
>>
>> Thanks.
>>
>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>> > hi
>> >
>> > How to intigrate flume with cassandra by using logsandra plugin.
>> >
>> > thank you.
>>
>
>
Re: flume-cassandra
Posted by shekhar sharma <sh...@gmail.com>.
You can use develop your own customized sink which can do pre-processing in
flume and then send the events to cassandra
Regards,
Som
On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rc...@ebay.com> wrote:
> You may also want to check out
>
> https://github.com/btoddb/flume-ng-cassandra-sink
>
> Roy
>
>
>
> -----Original Message-----
> From: rektide [mailto:rektide@voodoowarez.com]
> Sent: Wednesday, January 23, 2013 10:24 AM
> To: user@flume.apache.org
> Subject: Re: flume-cassandra
>
> Have you tried https://github.com/thobbs/logsandra#configuration ? How
> did that go?
>
> Thanks.
>
> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
> > hi
> >
> > How to intigrate flume with cassandra by using logsandra plugin.
> >
> > thank you.
>
RE: flume-cassandra
Posted by "Camp, Roy" <rc...@ebay.com>.
You may also want to check out
https://github.com/btoddb/flume-ng-cassandra-sink
Roy
-----Original Message-----
From: rektide [mailto:rektide@voodoowarez.com]
Sent: Wednesday, January 23, 2013 10:24 AM
To: user@flume.apache.org
Subject: Re: flume-cassandra
Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
Thanks.
On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
> hi
>
> How to intigrate flume with cassandra by using logsandra plugin.
>
> thank you.
Re: flume-cassandra
Posted by rektide <re...@voodoowarez.com>.
Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
Thanks.
On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
> hi
>
> How to intigrate flume with cassandra by using logsandra plugin.
>
> thank you.