You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Marimuthu Ponnambalam (JIRA)" <ji...@apache.org> on 2014/06/18 10:22:05 UTC

[jira] [Created] (FLUME-2405) Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource

Marimuthu Ponnambalam created FLUME-2405:
--------------------------------------------

             Summary: Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource
                 Key: FLUME-2405
                 URL: https://issues.apache.org/jira/browse/FLUME-2405
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: v1.5.0, v1.4.0
            Reporter: Marimuthu Ponnambalam
            Priority: Critical
             Fix For: v1.5.0.1, v1.6.0, v1.5.0, v1.4.0


Flume stops receiving data on scribe source if there is an arbitrary size of data send from scribe, beyond 16MB.

TFrameTransport throws exception, due to frame size is more than 16 MB, which is default size on TFrameTransport.

This can be resolved by adding one more property "thriftFrameSize", which will  allow custom frame size be configured, which will be used to create TFrameTransport.

This happens while scribe lag is being cleared, scribe sends Frame size larger than 16MB. 

http://stackoverflow.com/questions/24150439/flume-scribesource-how-to-set-thrift-frame-size



2014-06-10 19:40:40,405 WARN org.apache.thrift.server.THsHaServer: Exception while invoking!
org.apache.thrift.transport.TTransportException: Frame size (23757404) larger than max length (16384000)!
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)



--
This message was sent by Atlassian JIRA
(v6.2#6252)