[jira] Commented: (AVRO-747) NettyTransceiver: release semaphores on close so that clients are not blocked.

["Bruno Dumon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992016#comment-12992016 ] 

Bruno Dumon commented on AVRO-747:
----------------------------------

A problem with my previous patch is that when it releases the semaphore, NettyTransceiver.transceive() will return null. I thought this was fine, since this was already done in case of the other catched exceptions. However, this will give problems further on (NPE in ByteBufferInputStream) since apparently the transceive method is not supposed to return null.

I'll attach a newer patch that throws an IOException instead.

BTW, releaseSemaphore is also called by NettyClientAvroHandler.exceptionCaught, we should probably pass on the exception reported there.

It wouldn't harm if someone who actually knows Netty had a look at this.

> NettyTransceiver: release semaphores on close so that clients are not blocked.
> ------------------------------------------------------------------------------
>
>                 Key: AVRO-747
>                 URL: https://issues.apache.org/jira/browse/AVRO-747
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.5.0
>            Reporter: Bruno Dumon
>         Attachments: netty-transceiver-release-semaphores-on-close-patch.txt
>
>
> I use Avro RPC with the NettyTransceiver.
> When I kill the server, often the client hangs, jstack shows the following:
> {noformat}
> "pool-6-thread-1" prio=10 tid=0x09fef000 nid=0x3382 waiting on condition [0x76fc7000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0xa1df2e40> (a java.util.concurrent.Semaphore$NonfairSync)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
>         at org.apache.avro.ipc.NettyTransceiver$CallFuture.get(NettyTransceiver.java:207)
>         at org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:137)
>         at org.apache.avro.ipc.Requestor.request(Requestor.java:123)
>         - locked <0xa20986c0> (a org.apache.avro.specific.SpecificRequestor)
>         at org.apache.avro.specific.SpecificRequestor.invoke(SpecificRequestor.java:52)
> ...
> {noformat}
> Not that this matters much, but the client application is written such that it discovers the available servers via ZooKeeper. When a server disappears, it calls close on the corresponding NettyTransceiver.
> I have adjusted the NettyTransceiver.close() method to release any remaining semaphores, the same as is done in the exceptionCaught method of the UpstreamHandler. This solves the problem for me.
> Alternatively, we could handle channel close events in handleUpstream(), but I'm not sure if Netty automatically reconnects if the server re-appears, in which case this wouldn't be a good idea. OTOH, if the server would never come back, client threads could hang forever?
> Patch in attachment, against svn r1064125.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira