You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/03/16 19:55:41 UTC
[jira] [Commented] (GIRAPH-1137) Remove channel probing from Netty
worker thread for credit-based flow-control
[ https://issues.apache.org/jira/browse/GIRAPH-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928741#comment-15928741 ]
ASF GitHub Bot commented on GIRAPH-1137:
----------------------------------------
GitHub user heslami opened a pull request:
https://github.com/apache/giraph/pull/26
[GIRAPH-1137] Remove channel probing from Netty worker thread for credit-based flow…
In credit-based flow-control, sometimes, client threads (one type of Netty worker threads used in Giraph) try to send requests to other workers. This is bad practice for Netty and can cause Netty to mark the execution as deadlock-prone (an example exception shown below). Client threads should only be responsible for sending ACK/NACK messages in response to requests, and they should do so by reuseing the channel from which they received the request. In the current implementation, client threads may try to send unsent/cached requests in credit-based flow control. Sending such requests should be delegated to other threads.
WARN 2017-03-08 06:06:22,104 [netty-client-worker-3] ....
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@2c455378(incomplete)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:383)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:343)
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:259)
at org.apache.giraph.utils.ProgressableUtils$ChannelFutureWaitable.waitFor(ProgressableUtils.java:461)
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:214)
at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:180)
at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:165)
at org.apache.giraph.utils.ProgressableUtils.awaitChannelFuture(ProgressableUtils.java:132)
at org.apache.giraph.comm.netty.NettyClient.getNextChannel(NettyClient.java:715)
at org.apache.giraph.comm.netty.NettyClient.writeRequestToChannel(NettyClient.java:799)
at org.apache.giraph.comm.netty.NettyClient.doSend(NettyClient.java:789)
at org.apache.giraph.comm.flow_control.CreditBasedFlowControl.trySendCachedRequests(CreditBasedFlowControl.java:515)
at org.apache.giraph.comm.flow_control.CreditBasedFlowControl.messageAckReceived(CreditBasedFlowControl.java:485)
at org.apache.giraph.comm.netty.NettyClient.messageReceived(NettyClient.java:840)
at org.apache.giraph.comm.netty.handler.ResponseClientHandler.channelRead(ResponseClientHandler.java:87)
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153)
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at org.apache.giraph.comm.netty.InboundByteCounter.channelRead(InboundByteCounter.java:89)
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:745)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heslami/giraph fix-credit-based
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/giraph/pull/26.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #26
----
commit 4c8186cc8097877d5af20ef054630a629caaa026
Author: Hassan Eslami <ha...@wirelessprv-10-193-225-240.near.illinois.edu>
Date: 2017-03-16T19:52:12Z
Remove channel probing from Netty worker thread for credit-based flow-control
Closes GIRAPH-1137
----
> Remove channel probing from Netty worker thread for credit-based flow-control
> -----------------------------------------------------------------------------
>
> Key: GIRAPH-1137
> URL: https://issues.apache.org/jira/browse/GIRAPH-1137
> Project: Giraph
> Issue Type: Bug
> Reporter: Hassan Eslami
> Assignee: Hassan Eslami
>
> In credit-based flow-control, sometimes, client threads (one type of Netty worker threads used in Giraph) try to send requests to other workers. This is bad practice for Netty and can cause Netty to mark the execution as deadlock-prone (an example exception shown below). Client threads should only be responsible for sending ACK/NACK messages in response to requests, and they should do so by reuseing the channel from which they received the request. In the current implementation, client threads may try to send unsent/cached requests in credit-based flow control. Sending such requests should be delegated to other threads.
> WARN 2017-03-08 06:06:22,104 [netty-client-worker-3] ....
> io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@2c455378(incomplete)
> at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:383)
> at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
> at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:343)
> at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:259)
> at org.apache.giraph.utils.ProgressableUtils$ChannelFutureWaitable.waitFor(ProgressableUtils.java:461)
> at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:214)
> at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:180)
> at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:165)
> at org.apache.giraph.utils.ProgressableUtils.awaitChannelFuture(ProgressableUtils.java:132)
> at org.apache.giraph.comm.netty.NettyClient.getNextChannel(NettyClient.java:715)
> at org.apache.giraph.comm.netty.NettyClient.writeRequestToChannel(NettyClient.java:799)
> at org.apache.giraph.comm.netty.NettyClient.doSend(NettyClient.java:789)
> at org.apache.giraph.comm.flow_control.CreditBasedFlowControl.trySendCachedRequests(CreditBasedFlowControl.java:515)
> at org.apache.giraph.comm.flow_control.CreditBasedFlowControl.messageAckReceived(CreditBasedFlowControl.java:485)
> at org.apache.giraph.comm.netty.NettyClient.messageReceived(NettyClient.java:840)
> at org.apache.giraph.comm.netty.handler.ResponseClientHandler.channelRead(ResponseClientHandler.java:87)
> at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153)
> at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
> at org.apache.giraph.comm.netty.InboundByteCounter.channelRead(InboundByteCounter.java:89)
> at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
> at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)