You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Chad Rhyner (JIRA)" <ji...@apache.org> on 2014/09/05 21:16:28 UTC
[jira] [Updated] (FLUME-2456) Unable to use FileChannel on Flume
1.5.0.1
[ https://issues.apache.org/jira/browse/FLUME-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chad Rhyner updated FLUME-2456:
-------------------------------
Description:
I am unable to send logs when I am using a FileChannel in Flume 1.5.0.1. Here is the stack trace that I am seeing:
java.lang.IllegalStateException: Channel closed [channel=rio]. Due to java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:336)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
at org.apache.flume.channel.file.Log.replay(Log.java:449)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
When I change the agent configuration to use the memory channel, the error goes away, and my logs get transferred successfully.
Source host configuration broken looks like this:
agent.channels = channel
agent.sinks = sink
agent.channels.channel.type = file
agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data
agent.sinks.sink.channel = channel
agent.sinks.sink.type = avro
agent.sinks.sink.hostname = sink_hostname
agent.sinks.sink.port = 40000
agent.sources = source1
agent.sources.source1.type = exec
agent.sources.source1.command = tail -F /var/log/app.log
agent.sources.source1.channels = channel
However, using the same configuration, I am able to get the logs to successfully transfer when I change the configuration to use a memory channel instead:
By adding this:
agent.channels.channel.type = memory
And removing the file channel configuration:
agent.channels.channel.type = file
agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data
> Unable to use FileChannel on Flume 1.5.0.1
> ------------------------------------------
>
> Key: FLUME-2456
> URL: https://issues.apache.org/jira/browse/FLUME-2456
> Project: Flume
> Issue Type: Bug
> Components: File Channel
> Affects Versions: v1.4.0
> Reporter: Chad Rhyner
>
> I am unable to send logs when I am using a FileChannel in Flume 1.5.0.1. Here is the stack trace that I am seeing:
> java.lang.IllegalStateException: Channel closed [channel=rio]. Due to java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
> at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
> at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:336)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
> at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
> at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
> at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
> at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
> at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
> at org.apache.flume.channel.file.Log.replay(Log.java:449)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
> at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ... 1 more
> When I change the agent configuration to use the memory channel, the error goes away, and my logs get transferred successfully.
> Source host configuration broken looks like this:
> agent.channels = channel
> agent.sinks = sink
> agent.channels.channel.type = file
> agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
> agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data
> agent.sinks.sink.channel = channel
> agent.sinks.sink.type = avro
> agent.sinks.sink.hostname = sink_hostname
> agent.sinks.sink.port = 40000
> agent.sources = source1
> agent.sources.source1.type = exec
> agent.sources.source1.command = tail -F /var/log/app.log
> agent.sources.source1.channels = channel
> However, using the same configuration, I am able to get the logs to successfully transfer when I change the configuration to use a memory channel instead:
> By adding this:
> agent.channels.channel.type = memory
> And removing the file channel configuration:
> agent.channels.channel.type = file
> agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
> agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)