You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rajesh Jain <rj...@gmail.com> on 2013/07/17 22:47:14 UTC
Error: Flume to HDFS Sink
I am getting a strange error when I am configuring Flume to write to a HDFS
Sink.
17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
- Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
at
org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
Do I need to change my configure file, here is my flume-conf.properties
Second question, the hdfs.path - does it have to be hdfs namenode syntax
I am new to both flume and hadoop/hdfs.
# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*
# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler =
com.dynatrace.diagnostics.btexport.flume.BtExportHandler
# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer =
com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder
agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer =
com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder
agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer =
com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder
agent1.sinks.NullSink.type = null
# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100
agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100
agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel
VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel
agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel
Re: Error: Flume to HDFS Sink
Posted by Rajesh Jain <rj...@gmail.com>.
hi Paul
This did the trick.
agent1.channels.NullChannel.type = memory
Thanks
On Wed, Jul 17, 2013 at 5:03 PM, Paul Chavez <
pchavez@verticalsearchworks.com> wrote:
> **
> You've specified a channel named NullChannel in the agent1.channels
> properties, but you don't define it anywhere in the configuration. You've
> done the same with the NullSink.
>
> You'll need to add something like this to the config:
>
> agent1.channels.NullChannel.type = memory
>
> agent1.sinks.NullSink.type = null
>
> Hope that helps,
> Paul
>
>
> ------------------------------
> *From:* Rajesh Jain [mailto:rjain15@gmail.com]
> *Sent:* Wednesday, July 17, 2013 1:47 PM
> *To:* user@flume.apache.org
> *Subject:* Error: Flume to HDFS Sink
>
> I am getting a strange error when I am configuring Flume to write to a
> HDFS Sink.
>
> 17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
> - Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Selector channel not found: *NullChannel*
> at
> org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
>
>
> Do I need to change my configure file, here is my flume-conf.properties
>
> Second question, the hdfs.path - does it have to be hdfs namenode syntax
>
> I am new to both flume and hadoop/hdfs.
>
>
>
> # Name the components on this agent
> agent1.sources = HTTPSource
> agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
> agent1.channels = PurePathChannel UserActionChannel VisitChannel *NullChannel*
>
> # Describe/configure HTTPSource
> agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.HTTPSource.port = 4321
> agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler
>
> # Describe sinks
> agent1.sinks.PurePathSink.type = hdfs
> agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
> agent1.sinks.PurePathSink.hdfs.fileType = DataStream
> agent1.sinks.PurePathSink.hdfs.filePrefix = export
> agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
> agent1.sinks.PurePathSink.hdfs.rollInterval = 120
> agent1.sinks.PurePathSink.hdfs.rollSize = 131072
> agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder
>
> agent1.sinks.UserActionSink.type = hdfs
> agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
> agent1.sinks.UserActionSink.hdfs.fileType = DataStream
> agent1.sinks.UserActionSink.hdfs.filePrefix = export
> agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
> agent1.sinks.UserActionSink.hdfs.rollInterval = 120
> agent1.sinks.UserActionSink.hdfs.rollSize = 131072
> agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder
>
> agent1.sinks.VisitSink.type = hdfs
> agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
> agent1.sinks.VisitSink.hdfs.fileType = DataStream
> agent1.sinks.VisitSink.hdfs.filePrefix = export
> agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
> agent1.sinks.VisitSink.hdfs.rollInterval = 120
> agent1.sinks.VisitSink.hdfs.rollSize = 131072
> agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder
>
> agent1.sinks.NullSink.type = null
>
> # Use a channel which buffers events in memory
> agent1.channels.PurePathChannel.type = memory
> agent1.channels.PurePathChannel.capacity = 1000
> agent1.channels.PurePathChannel.transactionCapactiy = 100
>
> agent1.channels.UserActionChannel.type = memory
> agent1.channels.UserActionChannel.capacity = 1000
> agent1.channels.UserActionChannel.transactionCapactiy = 100
>
> agent1.channels.VisitChannel.type = memory
> agent1.channels.VisitChannel.capacity = 1000
> agent1.channels.VisitChannel.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
> agent1.sinks.PurePathSink.channel = PurePathChannel
> agent1.sinks.UserActionSink.channel = UserActionChannel
> agent1.sinks.VisitSink.channel = VisitChannel
> agent1.sinks.NullSink.channel = NullChannel
>
> agent1.sources.HTTPSource.selector.type = multiplexing
> agent1.sources.HTTPSource.selector.header = btType
> agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
> agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
> agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
> agent1.sources.HTTPSource.selector.default = NullChannel
>
>
RE: Error: Flume to HDFS Sink
Posted by Paul Chavez <pc...@verticalsearchworks.com>.
You've specified a channel named NullChannel in the agent1.channels properties, but you don't define it anywhere in the configuration. You've done the same with the NullSink.
You'll need to add something like this to the config:
agent1.channels.NullChannel.type = memory
agent1.sinks.NullSink.type = null
Hope that helps,
Paul
________________________________
From: Rajesh Jain [mailto:rjain15@gmail.com]
Sent: Wednesday, July 17, 2013 1:47 PM
To: user@flume.apache.org
Subject: Error: Flume to HDFS Sink
I am getting a strange error when I am configuring Flume to write to a HDFS Sink.
17 Jul 2013 20:36:46,469 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204) - Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Selector channel not found: NullChannel
at org.apache.flume.channel.MultiplexingChannelSelector.getChannelListFromNames(MultiplexingChannelSelector.java:166)
Do I need to change my configure file, here is my flume-conf.properties
Second question, the hdfs.path - does it have to be hdfs namenode syntax
I am new to both flume and hadoop/hdfs.
# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler
# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder
agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder
agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder
agent1.sinks.NullSink.type = null
# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100
agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100
agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel
agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel