You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Dineshraja (JIRA)" <ji...@apache.org> on 2017/04/05 12:58:41 UTC
[jira] [Commented] (FLUME-2477) Flume file channel fails when the
source file is larger than 64MB
[ https://issues.apache.org/jira/browse/FLUME-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956799#comment-15956799 ]
Dineshraja commented on FLUME-2477:
-----------------------------------
I'm facing the similar kind of problem.Is there any work around for this?
> Flume file channel fails when the source file is larger than 64MB
> -----------------------------------------------------------------
>
> Key: FLUME-2477
> URL: https://issues.apache.org/jira/browse/FLUME-2477
> Project: Flume
> Issue Type: Bug
> Components: File Channel
> Affects Versions: 1.5.0.1
> Reporter: Jorge Alvarez
>
> Flume file channel fails when the source file is larger than 64MB
> I am wondering if there is a way to configure the
> CodedInputStream.setSizeLimit from flume configuration or if there is any
> other way around this.
> org.apache.flume.ChannelException: Take failed due to IO error
> [channel=file-channel]
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:541)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:350)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: com.google.protobuf.InvalidProtocolBufferException:
> Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:89)
> at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:720)
> at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:666)
> at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3437)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
> at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
> at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
> at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
> at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
> at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
> at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257)
> at org.apache.flume.channel.file.Put.readProtos(Put.java:98)
> at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204)
> at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:292)
> at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:436)
> at org.apache.flume.channel.file.Log.get(Log.java:580)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:538)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)