[jira] [Created] (FLUME-2477) Flume file channel fails when the source file is larger than 64MB

["Jorge Alvarez (JIRA)" <ji...@apache.org>]

Jorge Alvarez created FLUME-2477:
------------------------------------

             Summary: Flume file channel fails when the source file is larger than 64MB
                 Key: FLUME-2477
                 URL: https://issues.apache.org/jira/browse/FLUME-2477
             Project: Flume
          Issue Type: Bug
          Components: File Channel
    Affects Versions: v1.5.0.1
            Reporter: Jorge Alvarez


Flume file channel fails when the source file is larger than 64MB
I am wondering if there is a way to configure the
CodedInputStream.setSizeLimit from flume configuration or if there is any
other way around this.

org.apache.flume.ChannelException: Take failed due to IO error
[channel=file-channel]
	at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:541)
	at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
	at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:350)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use
CodedInputStream.setSizeLimit() to increase the size limit.
	at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:89)
	at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:720)
	at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:666)
	at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
	at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3437)
	at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
	at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
	at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
	at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257)
	at org.apache.flume.channel.file.Put.readProtos(Put.java:98)
	at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204)
	at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:292)
	at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:436)
	at org.apache.flume.channel.file.Log.get(Log.java:580)
	at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:538)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)