[jira] [Resolved] (FLUME-2405) Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource

["Mike Percy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Percy resolved FLUME-2405.
-------------------------------

    Resolution: Duplicate

Thanks for the patch! We rolled this into a combined patch with a related change in FLUME-2404.

> Flume stops working on arbitrary Frame Size more than 16 MB on ScribeSource
> ---------------------------------------------------------------------------
>
>                 Key: FLUME-2405
>                 URL: https://issues.apache.org/jira/browse/FLUME-2405
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.4.0, v1.5.0
>            Reporter: Marimuthu Ponnambalam
>            Priority: Critical
>              Labels: FrameSize, Scribe, Source
>         Attachments: FLUME-2405.patch, ScribeSource.java
>
>
> Flume stops receiving data on scribe source if there is an arbitrary size of data send from scribe, beyond 16MB.
> TFrameTransport throws exception, due to frame size is more than 16 MB, which is default size on TFrameTransport.
> This can be resolved by adding one more property "thriftFrameSize", which will  allow custom frame size be configured, which will be used to create TFrameTransport.
> This happens while scribe lag is being cleared, scribe sends Frame size larger than 16MB. 
> http://stackoverflow.com/questions/24150439/flume-scribesource-how-to-set-thrift-frame-size
> 2014-06-10 19:40:40,405 WARN org.apache.thrift.server.THsHaServer: Exception while invoking!
> org.apache.thrift.transport.TTransportException: Frame size (23757404) larger than max length (16384000)!
>     at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
>     at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>     at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>     at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>     at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>     at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>     at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
>     at org.apache.thrift.server.Invocation.run(Invocation.java:18)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:724)



--
This message was sent by Atlassian JIRA
(v6.2#6252)