You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "Keuntae Park (JIRA)" <ji...@apache.org> on 2014/03/07 06:30:44 UTC
[jira] [Updated] (TAJO-675) size of frameDecoder should be
increased
[ https://issues.apache.org/jira/browse/TAJO-675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keuntae Park updated TAJO-675:
------------------------------
Summary: size of frameDecoder should be increased (was: size of freamDecoder should be increased)
> size of frameDecoder should be increased
> ----------------------------------------
>
> Key: TAJO-675
> URL: https://issues.apache.org/jira/browse/TAJO-675
> Project: Tajo
> Issue Type: Improvement
> Reporter: Keuntae Park
>
> As Repartitioner::scheduleHashShuffledFetches re-optimizes the number of tasks, sometimes one task may fetch results of thousands of tasks.
> In my case, there is a case that one task fetches from 9240 tasks,
> and got following exception
> {noformat}
> 2014-03-07 13:29:24,240 ERROR rpc.AsyncRpcClient (AsyncRpcClient.java:exceptionCaught(220)) - skt-rf-01/50.1.103.1:28093,class org.apache.tajo.ipc.QueryMasterProtocol,Adjusted frame length exceeds 2097152: 7729393 - discarded
> org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 2097152: 7729393 - discarded
> at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:417)
> at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:405)
> at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:320)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2014-03-07 13:29:24,242 ERROR rpc.AsyncRpcClient (AsyncRpcClient.java:exceptionCaught(235)) - RPC Exception:org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 2097152: 7729393 - discarded
> {noformat}
> The reason seems that frameDecoder setting in getPipeline() of ProtoPipelineFactory.java is 2MB
> {noformat}
> public ChannelPipeline getPipeline() throws Exception {
> ChannelPipeline p = Channels.pipeline();
> p.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1048576*2, 0, 4,
> 0, 4));
> p.addLast("protobufDecoder", new ProtobufDecoder(defaultInstance));
> p.addLast("frameEncoder", new LengthFieldPrepender(4));
> p.addLast("protobufEncoder", new ProtobufEncoder());
> p.addLast("handler", handler);
> return p;
> }
> {noformat}
> I think temporal solution for the issue is just increasing the size.
> However, I'm not sure what the proper size is.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)