You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by kashif khan <dr...@gmail.com> on 2012/11/29 13:03:11 UTC
Error in Upload the log file into hdfs
Hi,
I am just struggling to learn the flume and doing some testing. I am
running two agents (agent, agent1). The agent used to upload the log data
into hdfs and agent1 used as logger. The configuration of two agents as:
agent.sources = tail
agent.channels = memoryChannel
agent.sinks = hdfs-clusterSink
agent.sources.tail.type = exec
agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
agent.sources.tail.channels = memoryChannel
agent.sinks.hdfs-clusterSink.
channel = memoryChannel
agent.sinks.hdfs-clusterSink.type = hdfs
agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
hadoop1.example.com/user/root/Test/
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.transactionCapacity=1000
agent.channels.memoryChannel.capacity = 100
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 44444
# Describe sink1
agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
I dont know why it does not upload the log file into hdfs. where I am doing
mistake . If anyone who have solution please let me know.
The log file as:
29 Nov 2012 11:49:13,046 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:13,050 INFO [main]
(org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
Node manager starting
29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
Configuration provider starting
29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
Post-validation flume configuration contains configuration for agents:
[agent, agent1]
29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
- Failed to load configuration data. Exception follows.
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
29 Nov 2012 11:49:22,157 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:22,161 INFO [main]
(org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent1
29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
Node manager starting
29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
Configuration provider starting
29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
Post-validation flume configuration contains configuration for agents:
[agent, agent1]
29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
(org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
Monitoried counter group for type: CHANNEL, name: channel1, registered
successfully.
29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
- created channel channel1
29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
(org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
of sink: sink1, type: logger
29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
- Starting new configuration:{
sourceRunners:{source1=EventDrivenSourceRunner: {
source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
sinkRunners:{sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82d counterGroup:{
name:null counters:{} } }}
channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
- Starting Channel channel1
29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
(org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
Component type: CHANNEL, name: channel1 started
29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
- Waiting for channel: channel1 to start. Sleeping for 500 ms
29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
- Starting Sink sink1
29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
- Starting Source source1
29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
(org.apache.flume.source.NetcatSource.start:147) - Source starting
29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
(org.apache.flume.source.NetcatSource.start:161) - Created
serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
29 Nov 2012 11:49:33,327 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
68 65 6C 6C 6F 0D hello. }
29 Nov 2012 11:49:42,328 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
29 Nov 2012 11:49:46,359 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
Many thanks
Best regards,
Re: Error in Upload the log file into hdfs
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Brock, thats right. Will do over the weekend. We should add these into the Userguide, too.
Cheers,
Alex
On Nov 29, 2012, at 4:54 PM, Brock Noland <br...@cloudera.com> wrote:
> Alex is right and our error message there needs much improvement. I have
> created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744
>
> On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <wget.null@gmail.com
>> wrote:
>
>> Hi,
>>
>> agent.channels.memoryChannel.transactionCapacity=1000
>>
>> Is wrong. You cant have equal or more transaction capacity in a channel as
>> configured capacity. Use the opposite, when you want to use it.
>> from our Guide:
>>
>> capacity NUM The max number of events stored in the
>> channel
>> transactionCapacity NUM The max number of events stored in the
>> channel per transaction
>>
>> Try this:
>>
>> agent.channels.memoryChannel.capacity = 1000
>> agent.channels.memoryChannel.transactionCapacity=10
>>
>> cheers
>> - Alex
>>
>>
>> On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am just struggling to learn the flume and doing some testing. I am
>>> running two agents (agent, agent1). The agent used to upload the log data
>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>
>>> agent.sources = tail
>>> agent.channels = memoryChannel
>>> agent.sinks = hdfs-clusterSink
>>>
>>> agent.sources.tail.type = exec
>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>> agent.sources.tail.channels = memoryChannel
>>>
>>> agent.sinks.hdfs-clusterSink.
>>> channel = memoryChannel
>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>> hadoop1.example.com/user/root/Test/
>>>
>>>
>>> agent.channels.memoryChannel.type = memory
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>> agent.channels.memoryChannel.capacity = 100
>>>
>>>
>>>
>>>
>>> agent1.sources = source1
>>> agent1.sinks = sink1
>>> agent1.channels = channel1
>>>
>>> # Describe/configure source1
>>> agent1.sources.source1.type = netcat
>>> agent1.sources.source1.bind = localhost
>>> agent1.sources.source1.port = 44444
>>>
>>> # Describe sink1
>>> agent1.sinks.sink1.type = logger
>>>
>>> # Use a channel which buffers events in memory
>>> agent1.channels.channel1.type = memory
>>> agent1.channels.channel1.capacity = 1000
>>> agent1.channels.channel1.transactionCapactiy = 100
>>>
>>> # Bind the source and sink to the channel
>>> agent1.sources.source1.channels = channel1
>>> agent1.sinks.sink1.channel = channel1
>>>
>>>
>>> I dont know why it does not upload the log file into hdfs. where I am
>> doing
>>> mistake . If anyone who have solution please let me know.
>>>
>>>
>>> The log file as:
>>>
>>>
>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 1
>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>>> Node manager starting
>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 10
>>> 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>>> Configuration provider starting
>>> 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>>> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>> - Added sinks: hdfs-clusterSink Agent: agent
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:sink1
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:sink1
>>> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>> - Added sinks: sink1 Agent: agent1
>>> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
>>> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>>> Post-validation flume configuration contains configuration for agents:
>>> [agent, agent1]
>>> 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>>> - Creating channels
>>> 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
>>> - Failed to load configuration data. Exception follows.
>>> java.lang.IllegalStateException
>>> at
>>> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>>> at
>>> org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
>>> at
>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>> at
>>>
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
>>> at
>>>
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
>>> at
>>>
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>>> at
>>>
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>>> at
>>>
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at
>>>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> at
>>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> at
>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> 29 Nov 2012 11:49:22,157 INFO [main]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 1
>>> 29 Nov 2012 11:49:22,161 INFO [main]
>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>> agent1
>>> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>>> Node manager starting
>>> 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 10
>>> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>>> Configuration provider starting
>>> 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>>> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>> - Added sinks: hdfs-clusterSink Agent: agent
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:sink1
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:sink1
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>> - Added sinks: sink1 Agent: agent1
>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>> - Processing:hdfs-clusterSink
>>> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>>> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>>> Post-validation flume configuration contains configuration for agents:
>>> [agent, agent1]
>>> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>>> - Creating channels
>>> 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
>>> (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
>>> Monitoried counter group for type: CHANNEL, name: channel1, registered
>>> successfully.
>>> 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
>>> - created channel channel1
>>> 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
>>> (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
>>> of sink: sink1, type: logger
>>> 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
>>> - Starting new configuration:{
>>> sourceRunners:{source1=EventDrivenSourceRunner: {
>>> source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
>>> sinkRunners:{sink1=SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
>>> name:null counters:{} } }}
>>> channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
>> channel1}} }
>>> 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
>>> - Starting Channel channel1
>>> 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
>>> (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
>>> Component type: CHANNEL, name: channel1 started
>>> 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
>>> - Waiting for channel: channel1 to start. Sleeping for 500 ms
>>> 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
>>> - Starting Sink sink1
>>> 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
>>> - Starting Source source1
>>> 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
>>> (org.apache.flume.source.NetcatSource.start:147) - Source starting
>>> 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
>>> (org.apache.flume.source.NetcatSource.start:161) - Created
>>> serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
>>> 29 Nov 2012 11:49:33,327 INFO
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>>> 68 65 6C 6C 6F 0D hello. }
>>> 29 Nov 2012 11:49:42,328 INFO
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>>> 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
>>> 29 Nov 2012 11:49:46,359 INFO
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>>> 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
>>>
>>>
>>> Many thanks
>>>
>>> Best regards,
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Error in Upload the log file into hdfs
Posted by Brock Noland <br...@cloudera.com>.
Alex is right and our error message there needs much improvement. I have
created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744
On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:
> Hi,
>
> agent.channels.memoryChannel.transactionCapacity=1000
>
> Is wrong. You cant have equal or more transaction capacity in a channel as
> configured capacity. Use the opposite, when you want to use it.
> from our Guide:
>
> capacity NUM The max number of events stored in the
> channel
> transactionCapacity NUM The max number of events stored in the
> channel per transaction
>
> Try this:
>
> agent.channels.memoryChannel.capacity = 1000
> agent.channels.memoryChannel.transactionCapacity=10
>
> cheers
> - Alex
>
>
> On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
>
> > Hi,
> >
> > I am just struggling to learn the flume and doing some testing. I am
> > running two agents (agent, agent1). The agent used to upload the log data
> > into hdfs and agent1 used as logger. The configuration of two agents as:
> >
> > agent.sources = tail
> > agent.channels = memoryChannel
> > agent.sinks = hdfs-clusterSink
> >
> > agent.sources.tail.type = exec
> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> > agent.sources.tail.channels = memoryChannel
> >
> > agent.sinks.hdfs-clusterSink.
> > channel = memoryChannel
> > agent.sinks.hdfs-clusterSink.type = hdfs
> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> > hadoop1.example.com/user/root/Test/
> >
> >
> > agent.channels.memoryChannel.type = memory
> > agent.channels.memoryChannel.transactionCapacity=1000
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> >
> >
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = netcat
> > agent1.sources.source1.bind = localhost
> > agent1.sources.source1.port = 44444
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = logger
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > I dont know why it does not upload the log file into hdfs. where I am
> doing
> > mistake . If anyone who have solution please let me know.
> >
> >
> > The log file as:
> >
> >
> > 29 Nov 2012 11:49:13,046 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:13,050 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> > Post-validation flume configuration contains configuration for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
> > - Failed to load configuration data. Exception follows.
> > java.lang.IllegalStateException
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> > at
> > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
> > at
> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> > at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
> > at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 29 Nov 2012 11:49:22,157 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:22,161 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
> agent1
> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> > Post-validation flume configuration contains configuration for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
> > Monitoried counter group for type: CHANNEL, name: channel1, registered
> > successfully.
> > 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
> > - created channel channel1
> > 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
> > (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
> > of sink: sink1, type: logger
> > 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
> > - Starting new configuration:{
> > sourceRunners:{source1=EventDrivenSourceRunner: {
> > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
> > sinkRunners:{sink1=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
> > name:null counters:{} } }}
> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
> channel1}} }
> > 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
> > - Starting Channel channel1
> > 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
> > Component type: CHANNEL, name: channel1 started
> > 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
> > - Waiting for channel: channel1 to start. Sleeping for 500 ms
> > 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
> > - Starting Sink sink1
> > 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
> > - Starting Source source1
> > 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:147) - Source starting
> > 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:161) - Created
> > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
> > 29 Nov 2012 11:49:33,327 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 68 65 6C 6C 6F 0D hello. }
> > 29 Nov 2012 11:49:42,328 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
> > 29 Nov 2012 11:49:46,359 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
> >
> >
> > Many thanks
> >
> > Best regards,
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Error in Upload the log file into hdfs
Posted by kashif khan <dr...@gmail.com>.
Thank very much Alex.
Reagrds
On Thu, Nov 29, 2012 at 7:14 PM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:
> Alex
Re: Error in Upload the log file into hdfs
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi,
As Brock wrote, in the upcoming stable release 1.3.0 we implemented SpoolDirectorySource, which will do that for you. The appropriate Jira:
https://issues.apache.org/jira/browse/FLUME-1425
Cheers,
Alex
On Nov 29, 2012, at 6:03 PM, Brock Noland <br...@cloudera.com> wrote:
> Yes Flume-1.3.0 supports a source called Spool Directory Source which does
> this.
>
> On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <dr...@gmail.com>wrote:
>
>> Thanks Alexander
>>
>> Yeah you right, I was not read the document properly, sorry for that. once
>> read the documents, I have done and now uploading the data.
>>
>> I have just question about Flume 1.3. (which to be release very soon) Is
>> it support to automatically detect the file in folder and upload into HDFS?
>>
>> Many thanks
>>
>> Best regards
>>
>>
>> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
>> wget.null@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>
>>> Is wrong. You cant have equal or more transaction capacity in a channel
>>> as configured capacity. Use the opposite, when you want to use it.
>>> from our Guide:
>>>
>>> capacity NUM The max number of events stored in the
>>> channel
>>> transactionCapacity NUM The max number of events stored in the
>>> channel per transaction
>>>
>>> Try this:
>>>
>>> agent.channels.memoryChannel.capacity = 1000
>>> agent.channels.memoryChannel.transactionCapacity=10
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am just struggling to learn the flume and doing some testing. I am
>>>> running two agents (agent, agent1). The agent used to upload the log
>>> data
>>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>>
>>>> agent.sources = tail
>>>> agent.channels = memoryChannel
>>>> agent.sinks = hdfs-clusterSink
>>>>
>>>> agent.sources.tail.type = exec
>>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>>> agent.sources.tail.channels = memoryChannel
>>>>
>>>> agent.sinks.hdfs-clusterSink.
>>>> channel = memoryChannel
>>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>>> hadoop1.example.com/user/root/Test/
>>>>
>>>>
>>>> agent.channels.memoryChannel.type = memory
>>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>> agent.channels.memoryChannel.capacity = 100
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sources = source1
>>>> agent1.sinks = sink1
>>>> agent1.channels = channel1
>>>>
>>>> # Describe/configure source1
>>>> agent1.sources.source1.type = netcat
>>>> agent1.sources.source1.bind = localhost
>>>> agent1.sources.source1.port = 44444
>>>>
>>>> # Describe sink1
>>>> agent1.sinks.sink1.type = logger
>>>>
>>>> # Use a channel which buffers events in memory
>>>> agent1.channels.channel1.type = memory
>>>> agent1.channels.channel1.capacity = 1000
>>>> agent1.channels.channel1.transactionCapactiy = 100
>>>>
>>>> # Bind the source and sink to the channel
>>>> agent1.sources.source1.channels = channel1
>>>> agent1.sinks.sink1.channel = channel1
>>>>
>>>>
>>>> I dont know why it does not upload the log file into hdfs. where I am
>>> doing
>>>> mistake . If anyone who have solution please let me know.
>>>>
>>>>
>>>> The log file as:
>>>>
>>>>
>>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 1
>>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>>> agent
>>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>>> -
>>>> Node manager starting
>>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 10
>>>> 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
>>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>>> -
>>>> Configuration provider starting
>>>> 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>>>> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>>> - Added sinks: hdfs-clusterSink Agent: agent
>>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:sink1
>>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:sink1
>>>> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>>> - Added sinks: sink1 Agent: agent1
>>>> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
>>>> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>>>> Post-validation flume configuration contains configuration for agents:
>>>> [agent, agent1]
>>>> 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>>>> - Creating channels
>>>> 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
>>>> - Failed to load configuration data. Exception follows.
>>>> java.lang.IllegalStateException
>>>> at
>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>>>> at
>>>> org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
>>>> at
>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>>> at
>>>>
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
>>>> at
>>>>
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
>>>> at
>>>>
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>>>> at
>>>>
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>>>> at
>>>>
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>> at
>>>>
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>> at
>>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>>> at
>>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>>> at
>>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>>> at
>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> 29 Nov 2012 11:49:22,157 INFO [main]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 1
>>>> 29 Nov 2012 11:49:22,161 INFO [main]
>>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>>> agent1
>>>> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>>> -
>>>> Node manager starting
>>>> 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 10
>>>> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
>>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>>> -
>>>> Configuration provider starting
>>>> 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>>>> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>>> - Added sinks: hdfs-clusterSink Agent: agent
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:sink1
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:sink1
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>>> - Added sinks: sink1 Agent: agent1
>>>> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>>>> - Processing:hdfs-clusterSink
>>>> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>>>> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>>>> Post-validation flume configuration contains configuration for agents:
>>>> [agent, agent1]
>>>> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>>>> - Creating channels
>>>> 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
>>>> (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
>>>> Monitoried counter group for type: CHANNEL, name: channel1, registered
>>>> successfully.
>>>> 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
>>>> - created channel channel1
>>>> 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
>>>> (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating
>>> instance
>>>> of sink: sink1, type: logger
>>>> 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
>>>> - Starting new configuration:{
>>>> sourceRunners:{source1=EventDrivenSourceRunner: {
>>>> source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
>>>> sinkRunners:{sink1=SinkRunner: {
>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
>>>> name:null counters:{} } }}
>>>> channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
>>> channel1}} }
>>>> 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
>>>> - Starting Channel channel1
>>>> 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
>>>> (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
>>>> Component type: CHANNEL, name: channel1 started
>>>> 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
>>>> - Waiting for channel: channel1 to start. Sleeping for 500 ms
>>>> 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
>>>> - Starting Sink sink1
>>>> 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
>>>>
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
>>>> - Starting Source source1
>>>> 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
>>>> (org.apache.flume.source.NetcatSource.start:147) - Source starting
>>>> 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
>>>> (org.apache.flume.source.NetcatSource.start:161) - Created
>>>> serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
>>>> 29 Nov 2012 11:49:33,327 INFO
>>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>>> body:
>>>> 68 65 6C 6C 6F 0D hello. }
>>>> 29 Nov 2012 11:49:42,328 INFO
>>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>>> body:
>>>> 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
>>>> 29 Nov 2012 11:49:46,359 INFO
>>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>>> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>>> body:
>>>> 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
>>>>
>>>>
>>>> Many thanks
>>>>
>>>> Best regards,
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>
>>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Error in Upload the log file into hdfs
Posted by Brock Noland <br...@cloudera.com>.
Yes Flume-1.3.0 supports a source called Spool Directory Source which does
this.
On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <dr...@gmail.com>wrote:
> Thanks Alexander
>
> Yeah you right, I was not read the document properly, sorry for that. once
> read the documents, I have done and now uploading the data.
>
> I have just question about Flume 1.3. (which to be release very soon) Is
> it support to automatically detect the file in folder and upload into HDFS?
>
> Many thanks
>
> Best regards
>
>
> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
> wget.null@gmail.com> wrote:
>
>> Hi,
>>
>> agent.channels.memoryChannel.transactionCapacity=1000
>>
>> Is wrong. You cant have equal or more transaction capacity in a channel
>> as configured capacity. Use the opposite, when you want to use it.
>> from our Guide:
>>
>> capacity NUM The max number of events stored in the
>> channel
>> transactionCapacity NUM The max number of events stored in the
>> channel per transaction
>>
>> Try this:
>>
>> agent.channels.memoryChannel.capacity = 1000
>> agent.channels.memoryChannel.transactionCapacity=10
>>
>> cheers
>> - Alex
>>
>>
>> On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am just struggling to learn the flume and doing some testing. I am
>> > running two agents (agent, agent1). The agent used to upload the log
>> data
>> > into hdfs and agent1 used as logger. The configuration of two agents as:
>> >
>> > agent.sources = tail
>> > agent.channels = memoryChannel
>> > agent.sinks = hdfs-clusterSink
>> >
>> > agent.sources.tail.type = exec
>> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>> > agent.sources.tail.channels = memoryChannel
>> >
>> > agent.sinks.hdfs-clusterSink.
>> > channel = memoryChannel
>> > agent.sinks.hdfs-clusterSink.type = hdfs
>> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>> > hadoop1.example.com/user/root/Test/
>> >
>> >
>> > agent.channels.memoryChannel.type = memory
>> > agent.channels.memoryChannel.transactionCapacity=1000
>> > agent.channels.memoryChannel.capacity = 100
>> >
>> >
>> >
>> >
>> > agent1.sources = source1
>> > agent1.sinks = sink1
>> > agent1.channels = channel1
>> >
>> > # Describe/configure source1
>> > agent1.sources.source1.type = netcat
>> > agent1.sources.source1.bind = localhost
>> > agent1.sources.source1.port = 44444
>> >
>> > # Describe sink1
>> > agent1.sinks.sink1.type = logger
>> >
>> > # Use a channel which buffers events in memory
>> > agent1.channels.channel1.type = memory
>> > agent1.channels.channel1.capacity = 1000
>> > agent1.channels.channel1.transactionCapactiy = 100
>> >
>> > # Bind the source and sink to the channel
>> > agent1.sources.source1.channels = channel1
>> > agent1.sinks.sink1.channel = channel1
>> >
>> >
>> > I dont know why it does not upload the log file into hdfs. where I am
>> doing
>> > mistake . If anyone who have solution please let me know.
>> >
>> >
>> > The log file as:
>> >
>> >
>> > 29 Nov 2012 11:49:13,046 INFO [main]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 1
>> > 29 Nov 2012 11:49:13,050 INFO [main]
>> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>> agent
>> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>> > Node manager starting
>> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 10
>> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
>> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>> > Configuration provider starting
>> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>> > - Added sinks: hdfs-clusterSink Agent: agent
>> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:sink1
>> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:sink1
>> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>> > - Added sinks: sink1 Agent: agent1
>> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
>> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>> > Post-validation flume configuration contains configuration for agents:
>> > [agent, agent1]
>> > 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>> > - Creating channels
>> > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
>> > - Failed to load configuration data. Exception follows.
>> > java.lang.IllegalStateException
>> > at
>> > com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>> > at
>> > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
>> > at
>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>> > at
>> >
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
>> > at
>> >
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
>> > at
>> >
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>> > at
>> >
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>> > at
>> >
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>> > at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> > at
>> >
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> > 29 Nov 2012 11:49:22,157 INFO [main]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 1
>> > 29 Nov 2012 11:49:22,161 INFO [main]
>> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>> agent1
>> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>> > Node manager starting
>> > 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 10
>> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
>> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>> > Configuration provider starting
>> > 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>> > - Added sinks: hdfs-clusterSink Agent: agent
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:sink1
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:sink1
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>> > - Added sinks: sink1 Agent: agent1
>> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
>> > - Processing:hdfs-clusterSink
>> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
>> > Post-validation flume configuration contains configuration for agents:
>> > [agent, agent1]
>> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
>> > - Creating channels
>> > 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
>> > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
>> > Monitoried counter group for type: CHANNEL, name: channel1, registered
>> > successfully.
>> > 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
>> > - created channel channel1
>> > 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
>> > (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating
>> instance
>> > of sink: sink1, type: logger
>> > 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
>> > - Starting new configuration:{
>> > sourceRunners:{source1=EventDrivenSourceRunner: {
>> > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
>> > sinkRunners:{sink1=SinkRunner: {
>> > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
>> > name:null counters:{} } }}
>> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
>> channel1}} }
>> > 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
>> > - Starting Channel channel1
>> > 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
>> > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
>> > Component type: CHANNEL, name: channel1 started
>> > 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
>> > - Waiting for channel: channel1 to start. Sleeping for 500 ms
>> > 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
>> > - Starting Sink sink1
>> > 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
>> > - Starting Source source1
>> > 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
>> > (org.apache.flume.source.NetcatSource.start:147) - Source starting
>> > 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
>> > (org.apache.flume.source.NetcatSource.start:161) - Created
>> > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
>> > 29 Nov 2012 11:49:33,327 INFO
>> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>> > 68 65 6C 6C 6F 0D hello. }
>> > 29 Nov 2012 11:49:42,328 INFO
>> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>> > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
>> > 29 Nov 2012 11:49:46,359 INFO
>> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
>> body:
>> > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
>> >
>> >
>> > Many thanks
>> >
>> > Best regards,
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Error in Upload the log file into hdfs
Posted by kashif khan <dr...@gmail.com>.
Thanks Alexander
Yeah you right, I was not read the document properly, sorry for that. once
read the documents, I have done and now uploading the data.
I have just question about Flume 1.3. (which to be release very soon) Is
it support to automatically detect the file in folder and upload into HDFS?
Many thanks
Best regards
On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:
> Hi,
>
> agent.channels.memoryChannel.transactionCapacity=1000
>
> Is wrong. You cant have equal or more transaction capacity in a channel as
> configured capacity. Use the opposite, when you want to use it.
> from our Guide:
>
> capacity NUM The max number of events stored in the
> channel
> transactionCapacity NUM The max number of events stored in the
> channel per transaction
>
> Try this:
>
> agent.channels.memoryChannel.capacity = 1000
> agent.channels.memoryChannel.transactionCapacity=10
>
> cheers
> - Alex
>
>
> On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
>
> > Hi,
> >
> > I am just struggling to learn the flume and doing some testing. I am
> > running two agents (agent, agent1). The agent used to upload the log data
> > into hdfs and agent1 used as logger. The configuration of two agents as:
> >
> > agent.sources = tail
> > agent.channels = memoryChannel
> > agent.sinks = hdfs-clusterSink
> >
> > agent.sources.tail.type = exec
> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> > agent.sources.tail.channels = memoryChannel
> >
> > agent.sinks.hdfs-clusterSink.
> > channel = memoryChannel
> > agent.sinks.hdfs-clusterSink.type = hdfs
> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> > hadoop1.example.com/user/root/Test/
> >
> >
> > agent.channels.memoryChannel.type = memory
> > agent.channels.memoryChannel.transactionCapacity=1000
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> >
> >
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = netcat
> > agent1.sources.source1.bind = localhost
> > agent1.sources.source1.port = 44444
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = logger
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > I dont know why it does not upload the log file into hdfs. where I am
> doing
> > mistake . If anyone who have solution please let me know.
> >
> >
> > The log file as:
> >
> >
> > 29 Nov 2012 11:49:13,046 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:13,050 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> > Post-validation flume configuration contains configuration for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
> > - Failed to load configuration data. Exception follows.
> > java.lang.IllegalStateException
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> > at
> > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
> > at
> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> > at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
> > at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> > at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 29 Nov 2012 11:49:22,157 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:22,161 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
> agent1
> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> > Post-validation flume configuration contains configuration for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
> > Monitoried counter group for type: CHANNEL, name: channel1, registered
> > successfully.
> > 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
> > - created channel channel1
> > 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
> > (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
> > of sink: sink1, type: logger
> > 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
> > - Starting new configuration:{
> > sourceRunners:{source1=EventDrivenSourceRunner: {
> > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
> > sinkRunners:{sink1=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
> > name:null counters:{} } }}
> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
> channel1}} }
> > 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
> > - Starting Channel channel1
> > 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
> > Component type: CHANNEL, name: channel1 started
> > 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
> > - Waiting for channel: channel1 to start. Sleeping for 500 ms
> > 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
> > - Starting Sink sink1
> > 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
> > - Starting Source source1
> > 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:147) - Source starting
> > 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:161) - Created
> > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
> > 29 Nov 2012 11:49:33,327 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 68 65 6C 6C 6F 0D hello. }
> > 29 Nov 2012 11:49:42,328 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
> > 29 Nov 2012 11:49:46,359 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{}
> body:
> > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
> >
> >
> > Many thanks
> >
> > Best regards,
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
Re: Error in Upload the log file into hdfs
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi,
agent.channels.memoryChannel.transactionCapacity=1000
Is wrong. You cant have equal or more transaction capacity in a channel as configured capacity. Use the opposite, when you want to use it.
from our Guide:
capacity NUM The max number of events stored in the channel
transactionCapacity NUM The max number of events stored in the channel per transaction
Try this:
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity=10
cheers
- Alex
On Nov 29, 2012, at 1:03 PM, kashif khan <dr...@gmail.com> wrote:
> Hi,
>
> I am just struggling to learn the flume and doing some testing. I am
> running two agents (agent, agent1). The agent used to upload the log data
> into hdfs and agent1 used as logger. The configuration of two agents as:
>
> agent.sources = tail
> agent.channels = memoryChannel
> agent.sinks = hdfs-clusterSink
>
> agent.sources.tail.type = exec
> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> agent.sources.tail.channels = memoryChannel
>
> agent.sinks.hdfs-clusterSink.
> channel = memoryChannel
> agent.sinks.hdfs-clusterSink.type = hdfs
> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> hadoop1.example.com/user/root/Test/
>
>
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.transactionCapacity=1000
> agent.channels.memoryChannel.capacity = 100
>
>
>
>
> agent1.sources = source1
> agent1.sinks = sink1
> agent1.channels = channel1
>
> # Describe/configure source1
> agent1.sources.source1.type = netcat
> agent1.sources.source1.bind = localhost
> agent1.sources.source1.port = 44444
>
> # Describe sink1
> agent1.sinks.sink1.type = logger
>
> # Use a channel which buffers events in memory
> agent1.channels.channel1.type = memory
> agent1.channels.channel1.capacity = 1000
> agent1.channels.channel1.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.source1.channels = channel1
> agent1.sinks.sink1.channel = channel1
>
>
> I dont know why it does not upload the log file into hdfs. where I am doing
> mistake . If anyone who have solution please let me know.
>
>
> The log file as:
>
>
> 29 Nov 2012 11:49:13,046 INFO [main]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 1
> 29 Nov 2012 11:49:13,050 INFO [main]
> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
> Node manager starting
> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 10
> 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
> Configuration provider starting
> 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> - Added sinks: hdfs-clusterSink Agent: agent
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> - Added sinks: sink1 Agent: agent1
> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> Post-validation flume configuration contains configuration for agents:
> [agent, agent1]
> 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> - Creating channels
> 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
> - Failed to load configuration data. Exception follows.
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
> at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
> at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 29 Nov 2012 11:49:22,157 INFO [main]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 1
> 29 Nov 2012 11:49:22,161 INFO [main]
> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent1
> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
> Node manager starting
> 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 10
> 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
> Configuration provider starting
> 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> - Added sinks: hdfs-clusterSink Agent: agent
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> - Added sinks: sink1 Agent: agent1
> 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
> Post-validation flume configuration contains configuration for agents:
> [agent, agent1]
> 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0]
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> - Creating channels
> 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) -
> Monitoried counter group for type: CHANNEL, name: channel1, registered
> successfully.
> 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0]
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
> - created channel channel1
> 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0]
> (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance
> of sink: sink1, type: logger
> 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
> - Starting new configuration:{
> sourceRunners:{source1=EventDrivenSourceRunner: {
> source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
> sinkRunners:{sink1=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82d counterGroup:{
> name:null counters:{} } }}
> channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
> 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
> - Starting Channel channel1
> 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) -
> Component type: CHANNEL, name: channel1 started
> 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
> - Waiting for channel: channel1 to start. Sleeping for 500 ms
> 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
> - Starting Sink sink1
> 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
> - Starting Source source1
> 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3]
> (org.apache.flume.source.NetcatSource.start:147) - Source starting
> 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3]
> (org.apache.flume.source.NetcatSource.start:161) - Created
> serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
> 29 Nov 2012 11:49:33,327 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
> 68 65 6C 6C 6F 0D hello. }
> 29 Nov 2012 11:49:42,328 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
> 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. }
> 29 Nov 2012 11:49:46,359 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body:
> 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
>
>
> Many thanks
>
> Best regards,
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF