You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by David Capwell <dc...@gmail.com> on 2012/08/03 01:19:04 UTC
unable to write to HDFS with flume 1.2.0
I downloaded the latest release (1.2.0) of flume and tried to get it
sending data to HDFS; for some reason it isn't adding hdfs as a sink. Here
is my properties file:
agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = hdfsSink
# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channels = memoryChannel
agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
agent.sinks.hdfsSink.hdfs.filePrefix =
events-#agent.sinks.hdfsSink.hdfs.round = true
agent.sinks.hdfsSink.hdfs.roundValue =
10#agent.sinks.hdfsSink.hdfs.roundUnit = second
Here is what I see in the logs with debugging enabled:
2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
649 Creating sink: hdfsSink using HDFS
2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)]
668 Configuration empty for: hdfsSink.Removed.
Going over the code I wasn't sure where the ConfigurationException was
thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to
get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get
the same error.
am I missing something?
Thanks for your time reading this email.
Re: unable to write to HDFS with flume 1.2.0
Posted by David Capwell <dc...@gmail.com>.
Thanks! I have everything working just fine now with the following
settings (writing to s3)
agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = hdfsSink
# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = memoryChannel
agent.sources.seqGenSrc.interceptors = a b
agent.sources.seqGenSrc.interceptors.a.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.seqGenSrc.interceptors.a.preserveExisting = false
agent.sources.seqGenSrc.interceptors.a.hostHeader = hostname
agent.sources.seqGenSrc.interceptors.b.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity = 1000
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H
agent.sinks.hdfsSink.hdfs.filePrefix = events-
agent.sinks.hdfsSink.hdfs.batchSize = 100000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.rollSize = 0
agent.sinks.hdfsSink.hdfs.round = true
agent.sinks.hdfsSink.hdfs.roundValue = 10
agent.sinks.hdfsSink.hdfs.roundUnit = minute
On Thu, Aug 2, 2012 at 4:21 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:
> agent.sinks.hdfsSink.channels = memoryChannel <-needs to be
> "agent.sinks.hdfsSink.channel = memoryChannel"
>
> Hari
> --
> Hari Shreedharan
>
> On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote:
>
> I downloaded the latest release (1.2.0) of flume and tried to get it
> sending data to HDFS; for some reason it isn't adding hdfs as a sink. Here
> is my properties file:
>
>
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = hdfsSink
>
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
> agent.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
> agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 100
>
> agent.sinks.hdfsSink.type = hdfs
>
> agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
> agent.sinks.hdfsSink.hdfs.filePrefix =
> events-#agent.sinks.hdfsSink.hdfs.round = true
> agent.sinks.hdfsSink.hdfs.roundValue =
> 10#agent.sinks.hdfsSink.hdfs.roundUnit = second
>
>
> Here is what I see in the logs with debugging enabled:
>
> 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
> 649 Creating sink: hdfsSink using HDFS
> 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)]
> 668 Configuration empty for: hdfsSink.Removed.
>
> Going over the code I wasn't sure where the ConfigurationException was
> thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to
> get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get
> the same error.
>
>
> am I missing something?
>
> Thanks for your time reading this email.
>
>
>
Re: unable to write to HDFS with flume 1.2.0
Posted by Hari Shreedharan <hs...@cloudera.com>.
agent.sinks.hdfsSink.channels = memoryChannel <-needs to be "agent.sinks.hdfsSink.channel = memoryChannel"
Hari
--
Hari Shreedharan
On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote:
> I downloaded the latest release (1.2.0) of flume and tried to get it sending data to HDFS; for some reason it isn't adding hdfs as a sink. Here is my properties file:
>
>
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = hdfsSink
>
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
> agent.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
> agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 100
>
> agent.sinks.hdfsSink.type = hdfs
>
> agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
> agent.sinks.hdfsSink.hdfs.filePrefix = events-#agent.sinks.hdfsSink.hdfs.round = true
> agent.sinks.hdfsSink.hdfs.roundValue = 10#agent.sinks.hdfsSink.hdfs.roundUnit = second
>
>
>
> Here is what I see in the logs with debugging enabled:
>
> 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] 649 Creating sink: hdfsSink using HDFS
> 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] 668 Configuration empty for: hdfsSink.Removed.
>
>
> Going over the code I wasn't sure where the ConfigurationException was thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get the same error.
>
>
> am I missing something?
>
> Thanks for your time reading this email.