You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by kashif khan <dr...@gmail.com> on 2012/11/29 13:01:15 UTC
Error in Upload the flume log into HDFS
Hi,
I am just struggling to learn the flume and doing my some testing. I am
running two agents (agent, agent1). The agent used to upload the log data
into hdfs and agent1 used as logger. The configuration of two agents as:
agent.sources = tail
agent.channels = memoryChannel
agent.sinks = hdfs-clusterSink
agent.sources.tail.type = exec
agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
agent.sources.tail.channels = memoryChannel
agent.sinks.hdfs-clusterSink.channel = memoryChannel
agent.sinks.hdfs-clusterSink.type = hdfs
agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
hadoop1.example.com/user/root/Test/
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.transactionCapacity=1000
agent.channels.memoryChannel.capacity = 100
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 44444
# Describe sink1
agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
I dont know why it does not upload the log file into hdfs. where I am doing
mistake . If anyone who have solution please let me know.
The log file as:
29 Nov 2012 11:49:13,046 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:13,050 INFO [main]
(org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
Node manager starting
29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
Configuration provider starting
29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
Post-validation flume configuration contains configuration for agents:
[agent, agent1]
29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
- Failed to load configuration data. Exception follows.
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
29 Nov 2012 11:49:22,157 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:22,161 INFO [main]
(org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent1
29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
Node manager starting
29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
Configuration provider starting
29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
Post-validation flume configuration contains configuration for agents:
[agent, agent1]
29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
(org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
Monitoried counter group for type: CHANNEL, name: channel1, registered
successfully.
29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
- created channel channel1
29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
(org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
of sink: sink1, type: logger
29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
- Starting new configuration:{
sourceRunners:{source1=EventDrivenSourceRunner: {
source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
sinkRunners:{sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82d counterGroup:{
name:null counters:{} } }}
channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
- Starting Channel channel1
29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
(org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
Component type: CHANNEL, name: channel1 started
29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
- Waiting for channel: channel1 to start. Sleeping for 500 ms
29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
- Starting Sink sink1
29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
- Starting Source source1
29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
(org.apache.flume.source.NetcatSource.start:147) - Source starting
29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
(org.apache.flume.source.NetcatSource.start:161) - Created
serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
29 Nov 2012 11:49:33,327 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
68 65 6C 6C 6F 0D hello. }
29 Nov 2012 11:49:42,328 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
29 Nov 2012 11:49:46,359 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
Many thanks
Best regards,