You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jeff Bachtel <je...@gmail.com> on 2013/01/15 02:57:55 UTC
Unable to activate flume-ng-cassandra-sink
I have a build of Flume 1.4.0 on FreeBSD 8, installed into /usr/local/flume
. I manually created /usr/local/flume/plugins.d/flume-ng-cassandra-sink and
untarred flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
pull/build of btoddb's sources on github), so I have a lib/ directory
that's completely populated with both the sink plugin and any possible
jarfiles it might rely on.
I've likewise tried just copying flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar
to flume/lib/ for it to get picked up.
When I have a CassandraSink define in flume.conf as such:
webserver.sinks.cassandraSink.type =
com.tbtoddb.flume.sinks.cassandra.CassandraSink
webserver.sinks.cassandraSink.channel = memoryChannel
webserver.sinks.cassandraSink.hosts = localhost
webserver.sinks.cassandraSink.cluster-name = MyCluster
webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
webserver.sinks.cassandraSink.records-coldfam = Requests
and try to start Flume with:
/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
/usr/local/flume/conf/flume.conf -n webserver
-Dflume.root.logger=DEBUG,console
I get the following output (and obviously, no success):
2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type:
com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
com.tbtoddb.flume.sinks.cassandra.CassandraSink
at
org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
at
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
at
org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
at
org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException:
com.tbtoddb.flume.sinks.cassandra.CassandraSink
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at
org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
... 12 more
does anyone have any idea why this plugin isn't being picked up by Flume?
Is it an incompatibility with the plugin expecting to run under Flume
1.3.1, perhaps? I'm at a loss on debugging this.
Thanks,
Jeff
Re: Unable to activate flume-ng-cassandra-sink
Posted by Brock Noland <br...@cloudera.com>.
Hi,
We haven't documented it yet as it's not in any release. I've been
meaning to do that! :)
I just created https://issues.apache.org/jira/browse/FLUME-1845 for this.
Brock
On Tue, Jan 15, 2013 at 9:12 AM, Jeff Bachtel <je...@gmail.com> wrote:
> No, I hadn't separated the jarfiles out in that manner (I didn't realize the
> order they appeared in the CLASSPATH mattered). I'll certainly go change
> that now.
>
> Is there documentation on the plugins.d directory anywhere? I discovered it
> in the code by accident, and it met my needs while using this custom sink.
>
> Jeff
>
>
> On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>> Great to hear the plugin directory is being used. Just to confirm, do
>> you have the following:
>>
>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
>>
>> correct?
>>
>> Generally the plugin.d was created so this:
>>
>> /usr/local/flume/plugins.d/plugin/lib/
>>
>> would have the plugin jar and this:
>>
>> /usr/local/flume/plugins.d/plugin/libext/
>>
>> would have any jars the plugin depended on.
>>
>> Brock
>>
>> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <cw...@gmail.com>
>> wrote:
>> > The reason I don't recommend using it is that I didn't know it existed
>> > :P
>> >
>> > That seems like a great place to put it. I've done quite a bit of
>> > customization, so for me it's easier to just point Flume to the ./target
>> > directory of my java projects. But the plugins.d seems like a great
>> > place
>> > for third-party libraries.
>> >
>> > Glad I could help!
>> >
>> > - Connor
>> >
>> >
>> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <je...@gmail.com>
>> > wrote:
>> >>
>> >> Awesome, thanks; it was that typo.
>> >>
>> >> The plugins.d directory seems to be working, is there a particular
>> >> reason
>> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm
>> >> curious.
>> >>
>> >> Thanks again,
>> >>
>> >> Jeff
>> >>
>> >>
>> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson
>> >> <cw...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Leave the Cassandra sink jar with its other jars in a folder some
>> >>> where.
>> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH
>> >>> line.
>> >>> Uncomment it if needed. Append the path to your cassandra sink and its
>> >>> associated jars, and see if that works.
>> >>>
>> >>> There doesn't appear to be a version incompatability; simply, the
>> >>> Cassandra jar is not found in the classpath. And the reason for that
>> >>> is it
>> >>> looks like you have a typo in your flume.conf.
>> >>>
>> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> should probably be
>> >>>
>> >>> com.btoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> - Connor
>> >>>
>> >>>
>> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>> >>>> /usr/local/flume . I manually created
>> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>> >>>> pull/build
>> >>>> of btoddb's sources on github), so I have a lib/ directory that's
>> >>>> completely
>> >>>> populated with both the sink plugin and any possible jarfiles it
>> >>>> might rely
>> >>>> on.
>> >>>>
>> >>>> I've likewise tried just copying
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
>> >>>> get
>> >>>> picked up.
>> >>>>
>> >>>> When I have a CassandraSink define in flume.conf as such:
>> >>>> webserver.sinks.cassandraSink.type =
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> webserver.sinks.cassandraSink.channel = memoryChannel
>> >>>> webserver.sinks.cassandraSink.hosts = localhost
>> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests
>> >>>>
>> >>>> and try to start Flume with:
>> >>>>
>> >>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
>> >>>> /usr/local/flume/conf/flume.conf -n webserver
>> >>>> -Dflume.root.logger=DEBUG,console
>> >>>>
>> >>>>
>> >>>> I get the following output (and obviously, no success):
>> >>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
>> >>>>
>> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
>> >>>> Failed to load configuration data. Exception follows.
>> >>>> org.apache.flume.FlumeException: Unable to load sink type:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> >>>> at
>> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >>>> at
>> >>>>
>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >>>> at java.lang.Thread.run(Thread.java:722)
>> >>>> Caused by: java.lang.ClassNotFoundException:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> >>>> at java.security.AccessController.doPrivileged(Native Method)
>> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>> >>>> at
>> >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>> >>>> at java.lang.Class.forName0(Native Method)
>> >>>> at java.lang.Class.forName(Class.java:186)
>> >>>> at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>> >>>> ... 12 more
>> >>>>
>> >>>>
>> >>>> does anyone have any idea why this plugin isn't being picked up by
>> >>>> Flume? Is it an incompatibility with the plugin expecting to run
>> >>>> under Flume
>> >>>> 1.3.1, perhaps? I'm at a loss on debugging this.
>> >>>>
>> >>>> Thanks,
>> >>>> Jeff
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Unable to activate flume-ng-cassandra-sink
Posted by Jeff Bachtel <je...@gmail.com>.
No, I hadn't separated the jarfiles out in that manner (I didn't realize
the order they appeared in the CLASSPATH mattered). I'll certainly go
change that now.
Is there documentation on the plugins.d directory anywhere? I discovered it
in the code by accident, and it met my needs while using this custom sink.
Jeff
On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <br...@cloudera.com> wrote:
> Great to hear the plugin directory is being used. Just to confirm, do
> you have the following:
>
> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
>
> correct?
>
> Generally the plugin.d was created so this:
>
> /usr/local/flume/plugins.d/plugin/lib/
>
> would have the plugin jar and this:
>
> /usr/local/flume/plugins.d/plugin/libext/
>
> would have any jars the plugin depended on.
>
> Brock
>
> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <cw...@gmail.com>
> wrote:
> > The reason I don't recommend using it is that I didn't know it existed :P
> >
> > That seems like a great place to put it. I've done quite a bit of
> > customization, so for me it's easier to just point Flume to the ./target
> > directory of my java projects. But the plugins.d seems like a great place
> > for third-party libraries.
> >
> > Glad I could help!
> >
> > - Connor
> >
> >
> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <je...@gmail.com>
> > wrote:
> >>
> >> Awesome, thanks; it was that typo.
> >>
> >> The plugins.d directory seems to be working, is there a particular
> reason
> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm
> curious.
> >>
> >> Thanks again,
> >>
> >> Jeff
> >>
> >>
> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson <cwoodson.dev@gmail.com
> >
> >> wrote:
> >>>
> >>> Leave the Cassandra sink jar with its other jars in a folder some
> where.
> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH
> line.
> >>> Uncomment it if needed. Append the path to your cassandra sink and its
> >>> associated jars, and see if that works.
> >>>
> >>> There doesn't appear to be a version incompatability; simply, the
> >>> Cassandra jar is not found in the classpath. And the reason for that
> is it
> >>> looks like you have a typo in your flume.conf.
> >>>
> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> >>>
> >>> should probably be
> >>>
> >>> com.btoddb.flume.sinks.cassandra.CassandraSink
> >>>
> >>> - Connor
> >>>
> >>>
> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>
> >>> wrote:
> >>>>
> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
> >>>> /usr/local/flume . I manually created
> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
> pull/build
> >>>> of btoddb's sources on github), so I have a lib/ directory that's
> completely
> >>>> populated with both the sink plugin and any possible jarfiles it
> might rely
> >>>> on.
> >>>>
> >>>> I've likewise tried just copying
> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to get
> >>>> picked up.
> >>>>
> >>>> When I have a CassandraSink define in flume.conf as such:
> >>>> webserver.sinks.cassandraSink.type =
> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> >>>> webserver.sinks.cassandraSink.channel = memoryChannel
> >>>> webserver.sinks.cassandraSink.hosts = localhost
> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests
> >>>>
> >>>> and try to start Flume with:
> >>>>
> >>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
> >>>> /usr/local/flume/conf/flume.conf -n webserver
> >>>> -Dflume.root.logger=DEBUG,console
> >>>>
> >>>>
> >>>> I get the following output (and obviously, no success):
> >>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
> >>>>
> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
> >>>> Failed to load configuration data. Exception follows.
> >>>> org.apache.flume.FlumeException: Unable to load sink type:
> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> >>>> at
> >>>>
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
> >>>> at
> >>>>
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
> >>>> at
> >>>>
> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
> >>>> at
> >>>>
> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
> >>>> at
> >>>>
> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
> >>>> at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at
> >>>>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >>>> at
> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >>>> at
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >>>> at
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >>>> at
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>>> at java.lang.Thread.run(Thread.java:722)
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> >>>> at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> >>>> at java.lang.Class.forName0(Native Method)
> >>>> at java.lang.Class.forName(Class.java:186)
> >>>> at
> >>>>
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
> >>>> ... 12 more
> >>>>
> >>>>
> >>>> does anyone have any idea why this plugin isn't being picked up by
> >>>> Flume? Is it an incompatibility with the plugin expecting to run
> under Flume
> >>>> 1.3.1, perhaps? I'm at a loss on debugging this.
> >>>>
> >>>> Thanks,
> >>>> Jeff
> >>>
> >>>
> >>
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: Unable to activate flume-ng-cassandra-sink
Posted by Brock Noland <br...@cloudera.com>.
Great to hear the plugin directory is being used. Just to confirm, do
you have the following:
/usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
correct?
Generally the plugin.d was created so this:
/usr/local/flume/plugins.d/plugin/lib/
would have the plugin jar and this:
/usr/local/flume/plugins.d/plugin/libext/
would have any jars the plugin depended on.
Brock
On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <cw...@gmail.com> wrote:
> The reason I don't recommend using it is that I didn't know it existed :P
>
> That seems like a great place to put it. I've done quite a bit of
> customization, so for me it's easier to just point Flume to the ./target
> directory of my java projects. But the plugins.d seems like a great place
> for third-party libraries.
>
> Glad I could help!
>
> - Connor
>
>
> On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <je...@gmail.com>
> wrote:
>>
>> Awesome, thanks; it was that typo.
>>
>> The plugins.d directory seems to be working, is there a particular reason
>> you don't recommend using? I'm brand-spanking new to Flume, so I'm curious.
>>
>> Thanks again,
>>
>> Jeff
>>
>>
>> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson <cw...@gmail.com>
>> wrote:
>>>
>>> Leave the Cassandra sink jar with its other jars in a folder some where.
>>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH line.
>>> Uncomment it if needed. Append the path to your cassandra sink and its
>>> associated jars, and see if that works.
>>>
>>> There doesn't appear to be a version incompatability; simply, the
>>> Cassandra jar is not found in the classpath. And the reason for that is it
>>> looks like you have a typo in your flume.conf.
>>>
>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>>
>>> should probably be
>>>
>>> com.btoddb.flume.sinks.cassandra.CassandraSink
>>>
>>> - Connor
>>>
>>>
>>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>
>>> wrote:
>>>>
>>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>>>> /usr/local/flume . I manually created
>>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
>>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local pull/build
>>>> of btoddb's sources on github), so I have a lib/ directory that's completely
>>>> populated with both the sink plugin and any possible jarfiles it might rely
>>>> on.
>>>>
>>>> I've likewise tried just copying
>>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to get
>>>> picked up.
>>>>
>>>> When I have a CassandraSink define in flume.conf as such:
>>>> webserver.sinks.cassandraSink.type =
>>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>>> webserver.sinks.cassandraSink.channel = memoryChannel
>>>> webserver.sinks.cassandraSink.hosts = localhost
>>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>>>> webserver.sinks.cassandraSink.records-coldfam = Requests
>>>>
>>>> and try to start Flume with:
>>>>
>>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
>>>> /usr/local/flume/conf/flume.conf -n webserver
>>>> -Dflume.root.logger=DEBUG,console
>>>>
>>>>
>>>> I get the following output (and obviously, no success):
>>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
>>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
>>>> Failed to load configuration data. Exception follows.
>>>> org.apache.flume.FlumeException: Unable to load sink type:
>>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
>>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>>> at
>>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>>>> at
>>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>>>> at
>>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>>>> at
>>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>>>> at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> at java.lang.Thread.run(Thread.java:722)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>>>> at java.lang.Class.forName0(Native Method)
>>>> at java.lang.Class.forName(Class.java:186)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>>>> ... 12 more
>>>>
>>>>
>>>> does anyone have any idea why this plugin isn't being picked up by
>>>> Flume? Is it an incompatibility with the plugin expecting to run under Flume
>>>> 1.3.1, perhaps? I'm at a loss on debugging this.
>>>>
>>>> Thanks,
>>>> Jeff
>>>
>>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Unable to activate flume-ng-cassandra-sink
Posted by Connor Woodson <cw...@gmail.com>.
The reason I don't recommend using it is that I didn't know it existed :P
That seems like a great place to put it. I've done quite a bit of
customization, so for me it's easier to just point Flume to the ./target
directory of my java projects. But the plugins.d seems like a great place
for third-party libraries.
Glad I could help!
- Connor
On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <je...@gmail.com>wrote:
> Awesome, thanks; it was that typo.
>
> The plugins.d directory seems to be working, is there a particular reason
> you don't recommend using? I'm brand-spanking new to Flume, so I'm curious.
>
> Thanks again,
>
> Jeff
>
>
> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson <cw...@gmail.com>wrote:
>
>> Leave the Cassandra sink jar with its other jars in a folder some where.
>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH line.
>> Uncomment it if needed. Append the path to your cassandra sink and its
>> associated jars, and see if that works.
>>
>> There doesn't appear to be a version incompatability; simply, the
>> Cassandra jar is not found in the classpath. And the reason for that is it
>> looks like you have a typo in your flume.conf.
>>
>> com.*t*btoddb.flume.sinks.cassandra.CassandraSink
>>
>> should probably be
>>
>> com.btoddb.flume.sinks.cassandra.CassandraSink
>>
>> - Connor
>>
>>
>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>wrote:
>>
>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>>> /usr/local/flume . I manually created
>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and
>>> untarred flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>>> pull/build of btoddb's sources on github), so I have a lib/ directory
>>> that's completely populated with both the sink plugin and any possible
>>> jarfiles it might rely on.
>>>
>>> I've likewise tried just
>>> copying flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
>>> get picked up.
>>>
>>> When I have a CassandraSink define in flume.conf as such:
>>> webserver.sinks.cassandraSink.type =
>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>> webserver.sinks.cassandraSink.channel = memoryChannel
>>> webserver.sinks.cassandraSink.hosts = localhost
>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>>> webserver.sinks.cassandraSink.records-coldfam = Requests
>>>
>>> and try to start Flume with:
>>>
>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
>>> /usr/local/flume/conf/flume.conf -n webserver
>>> -Dflume.root.logger=DEBUG,console
>>>
>>>
>>> I get the following output (and obviously, no success):
>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
>>> Failed to load configuration data. Exception follows.
>>> org.apache.flume.FlumeException: Unable to load sink type:
>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>>> at
>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:186)
>>> at
>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>>> ... 12 more
>>>
>>>
>>> does anyone have any idea why this plugin isn't being picked up by
>>> Flume? Is it an incompatibility with the plugin expecting to run under
>>> Flume 1.3.1, perhaps? I'm at a loss on debugging this.
>>>
>>> Thanks,
>>> Jeff
>>>
>>
>>
>
Re: Unable to activate flume-ng-cassandra-sink
Posted by Jeff Bachtel <je...@gmail.com>.
Awesome, thanks; it was that typo.
The plugins.d directory seems to be working, is there a particular reason
you don't recommend using? I'm brand-spanking new to Flume, so I'm curious.
Thanks again,
Jeff
On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson <cw...@gmail.com>wrote:
> Leave the Cassandra sink jar with its other jars in a folder some where.
> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH line.
> Uncomment it if needed. Append the path to your cassandra sink and its
> associated jars, and see if that works.
>
> There doesn't appear to be a version incompatability; simply, the
> Cassandra jar is not found in the classpath. And the reason for that is it
> looks like you have a typo in your flume.conf.
>
> com.*t*btoddb.flume.sinks.cassandra.CassandraSink
>
> should probably be
>
> com.btoddb.flume.sinks.cassandra.CassandraSink
>
> - Connor
>
>
> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>wrote:
>
>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>> /usr/local/flume . I manually created
>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and
>> untarred flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>> pull/build of btoddb's sources on github), so I have a lib/ directory
>> that's completely populated with both the sink plugin and any possible
>> jarfiles it might rely on.
>>
>> I've likewise tried just
>> copying flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
>> get picked up.
>>
>> When I have a CassandraSink define in flume.conf as such:
>> webserver.sinks.cassandraSink.type =
>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> webserver.sinks.cassandraSink.channel = memoryChannel
>> webserver.sinks.cassandraSink.hosts = localhost
>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>> webserver.sinks.cassandraSink.records-coldfam = Requests
>>
>> and try to start Flume with:
>>
>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
>> /usr/local/flume/conf/flume.conf -n webserver
>> -Dflume.root.logger=DEBUG,console
>>
>>
>> I get the following output (and obviously, no success):
>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
>> Failed to load configuration data. Exception follows.
>> org.apache.flume.FlumeException: Unable to load sink type:
>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> at
>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>> at
>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>> at
>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.lang.ClassNotFoundException:
>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:186)
>> at
>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>> ... 12 more
>>
>>
>> does anyone have any idea why this plugin isn't being picked up by Flume?
>> Is it an incompatibility with the plugin expecting to run under Flume
>> 1.3.1, perhaps? I'm at a loss on debugging this.
>>
>> Thanks,
>> Jeff
>>
>
>
Re: Unable to activate flume-ng-cassandra-sink
Posted by Connor Woodson <cw...@gmail.com>.
Leave the Cassandra sink jar with its other jars in a folder some where. In
FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH line.
Uncomment it if needed. Append the path to your cassandra sink and its
associated jars, and see if that works.
There doesn't appear to be a version incompatability; simply, the Cassandra
jar is not found in the classpath. And the reason for that is it looks like
you have a typo in your flume.conf.
com.*t*btoddb.flume.sinks.cassandra.CassandraSink
should probably be
com.btoddb.flume.sinks.cassandra.CassandraSink
- Connor
On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <je...@gmail.com>wrote:
> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
> /usr/local/flume . I manually created
> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and
> untarred flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
> pull/build of btoddb's sources on github), so I have a lib/ directory
> that's completely populated with both the sink plugin and any possible
> jarfiles it might rely on.
>
> I've likewise tried just
> copying flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to
> get picked up.
>
> When I have a CassandraSink define in flume.conf as such:
> webserver.sinks.cassandraSink.type =
> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> webserver.sinks.cassandraSink.channel = memoryChannel
> webserver.sinks.cassandraSink.hosts = localhost
> webserver.sinks.cassandraSink.cluster-name = MyCluster
> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
> webserver.sinks.cassandraSink.records-coldfam = Requests
>
> and try to start Flume with:
>
> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
> /usr/local/flume/conf/flume.conf -n webserver
> -Dflume.root.logger=DEBUG,console
>
>
> I get the following output (and obviously, no success):
> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
> Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Unable to load sink type:
> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> at
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
> at
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
> at
> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
> at
> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
> at
> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.ClassNotFoundException:
> com.tbtoddb.flume.sinks.cassandra.CassandraSink
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:186)
> at
> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
> ... 12 more
>
>
> does anyone have any idea why this plugin isn't being picked up by Flume?
> Is it an incompatibility with the plugin expecting to run under Flume
> 1.3.1, perhaps? I'm at a loss on debugging this.
>
> Thanks,
> Jeff
>