You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Deeter, Derek" <de...@digitalinsight.com> on 2011/01/07 03:22:45 UTC
Newbie question - logsandra Flume Sink plugin.
Hi -
I'm not sure if this should be asked on the Cassandra or Flume list, so I'm trying both -
I am trying to do a proof of concept for logging into Cassandra. We need to capture large volumes of audit events in a non-lossy manner at the same time being able to quickly access those events for various reports, so this Flume-Cassandra combination seems to be just the ticket. For this POC, I've set up a ring/cluster of 4 Cassandra nodes and they are all running successfully it seems (version 0.6.8), and I can do the simple entry and access of data by hand using the cli.
Next I am trying to use Flume to push log data into Cassandra - I've set up the logsandra keyspace for logsandraSyslogSink and the extra column families for the simpleCassandraSink, set up the plugins for Flume, updated the flume-site.xml to add the plugin references, and set the $FLUME_CLASSPATH all as mentioned on the https://github.com/thobbs/flume-cassandra-plugin page. However, I always get an "Invalid sink" error when I try to configure a Flume node to have Cassandra as a sink. Neither of these commands work:
172.16.199.166 : collectorSource(35853) | simpleCassandraSink("KeySpace1","data","FlumeIndexes","172.16.200.130:7005");
172.16.199.166 : collectorSource(35853) | logsandraSyslogSink("172.16.200.130:7005");
Can anyone please tell me if this is the correct command, or what the format should be? It is not really mentioned in the instructions.
I do see the following when starting the flume nodes, but not on the flume master:
2011-01-06 17:12:06,586 [main] INFO conf.SinkFactoryImpl: Found sink builder simpleCassandraSink in org.apache.cassandra.plugins.SimpleCassandraSink
2011-01-06 17:12:06,586 [main] WARN conf.SinkFactoryImpl: No sink decorators found in org.apache.cassandra.plugins.SimpleCassandraSink
2011-01-06 17:12:06,586 [main] INFO conf.SinkFactoryImpl: Found sink builder logsandraSyslogSink in org.apache.cassandra.plugins.LogsandraSyslogSink
2011-01-06 17:12:06,586 [main] WARN conf.SinkFactoryImpl: No sink decorators found in org.apache.cassandra.plugins.LogsandraSyslogSink
So it looks like the plugins are being loaded - without going into the source I am at a loss at the 'Invalid Sink' error. Any information appreciated -
Thanks in advance,
-Derek
--
Derek Deeter, Sr. Software Engineer Intuit Financial Services
5601 Lindero Canyon Road.
derek.deeter@digitalinsight.com Westlake, CA 91362
Re: Newbie question - logsandra Flume Sink plugin.
Posted by Tyler Hobbs <ty...@riptano.com>.
Hi Derek,
The Cassandra sink currently only works with Cassandra 0.7, although I don't
suppose that is what is causing the 'InvalidSink' problem.
What version of Flume are you using? The plugin works with Flume 0.9.1, but
for 0.9.2 they upgraded the Thrift library version that the project uses and
I haven't had the time yet to update the plugin. This would normally be a
compile-time error, so I imagine 'InvalidSink' is Flume's response to that.
Adding support for Cassandra 0.6 and Flume 0.9.2 are both on my TODO list.
If either of those are hard version requirements for you, I could make the
effort sooner rather than later.
I'm glad you're checking it out! Let me know if you think of any
improvements once you get it working :)
- Tyler
On Thu, Jan 6, 2011 at 8:22 PM, Deeter, Derek <
derek.deeter@digitalinsight.com> wrote:
> Hi -
>
> I'm not sure if this should be asked on the Cassandra or Flume list, so I'm
> trying both -
>
> I am trying to do a proof of concept for logging into Cassandra. We need
> to capture large volumes of audit events in a non-lossy manner at the same
> time being able to quickly access those events for various reports, so this
> Flume-Cassandra combination seems to be just the ticket. For this POC, I've
> set up a ring/cluster of 4 Cassandra nodes and they are all running
> successfully it seems (version 0.6.8), and I can do the simple entry and
> access of data by hand using the cli.
>
> Next I am trying to use Flume to push log data into Cassandra - I've set up
> the logsandra keyspace for logsandraSyslogSink and the extra column families
> for the simpleCassandraSink, set up the plugins for Flume, updated the
> flume-site.xml to add the plugin references, and set the $FLUME_CLASSPATH
> all as mentioned on the https://github.com/thobbs/flume-cassandra-pluginpage. However, I always get an "Invalid sink" error when I try to configure
> a Flume node to have Cassandra as a sink. Neither of these commands work:
>
> 172.16.199.166 : collectorSource(35853) |
> simpleCassandraSink("KeySpace1","data","FlumeIndexes","172.16.200.130:7005
> ");
> 172.16.199.166 : collectorSource(35853) | logsandraSyslogSink("
> 172.16.200.130:7005");
>
> Can anyone please tell me if this is the correct command, or what the
> format should be? It is not really mentioned in the instructions.
>
> I do see the following when starting the flume nodes, but not on the flume
> master:
> 2011-01-06 17:12:06,586 [main] INFO conf.SinkFactoryImpl: Found sink
> builder simpleCassandraSink in
> org.apache.cassandra.plugins.SimpleCassandraSink
> 2011-01-06 17:12:06,586 [main] WARN conf.SinkFactoryImpl: No sink
> decorators found in org.apache.cassandra.plugins.SimpleCassandraSink
> 2011-01-06 17:12:06,586 [main] INFO conf.SinkFactoryImpl: Found sink
> builder logsandraSyslogSink in
> org.apache.cassandra.plugins.LogsandraSyslogSink
> 2011-01-06 17:12:06,586 [main] WARN conf.SinkFactoryImpl: No sink
> decorators found in org.apache.cassandra.plugins.LogsandraSyslogSink
>
> So it looks like the plugins are being loaded - without going into the
> source I am at a loss at the 'Invalid Sink' error. Any information
> appreciated -
>
> Thanks in advance,
> -Derek
>
>
> --
> Derek Deeter, Sr. Software Engineer Intuit Financial Services
> 5601 Lindero Canyon
> Road.
> derek.deeter@digitalinsight.com Westlake, CA 91362
>
>
>