You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Himanshu Patidar <hi...@hotmail.com> on 2014/04/22 11:29:49 UTC
Maximum size of the request accepted by Avro Source
Hi,
I have a flume agent with Avro Source, memory channel and a custom sink. I am trying to send a single event with the size of 10MBs but I am getting GC overhead and Java out of memory errors. (Currently the heap size is 4GBs). Can anyone suggest me what can be the maximum size of the request that a flume Avro Source accepts.
Thanks,Himanshu
Re: Maximum size of the request accepted by Avro Source
Posted by Ashish <pa...@gmail.com>.
This is bound to happen for large file. Avro's Netty implementation uses
FrameDecoder, which keep the chunks in memory till the data is complete for
decoding. So for larger files OOME shall happen sooner or later. The max
size is more related to memory available than a fixed number. More number
of concurrent connection uploading would result in OOME faster.
The solution is to use a decoder that can push data to file system, rather
than keeping in memory.
There shall be similar implementation or you may need to write. This change
needs to be done in Avro code base, check org.apache.avro.ipc.NettyServer
Line#97 (for Avro 1.7.3)
thanks
ashish
On Tue, Apr 22, 2014 at 2:59 PM, Himanshu Patidar <
himanshu.patidar@hotmail.com> wrote:
> Hi,
>
> I have a flume agent with Avro Source, memory channel and a custom sink. I
> am trying to send a single event with the size of 10MBs but I am getting GC
> overhead and Java out of memory errors. (Currently the heap size is 4GBs).
> Can anyone suggest me what can be the maximum size of the request that a
> flume Avro Source accepts.
>
> Thanks,
> Himanshu
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal