You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/05/23 22:22:00 UTC
[jira] [Commented] (KUDU-3277) unexpected exception from downstream
with kudu client 1.10
[ https://issues.apache.org/jira/browse/KUDU-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350147#comment-17350147 ]
ASF subversion and git services commented on KUDU-3277:
-------------------------------------------------------
Commit d1969e2623e30bfd62f2463f15e0bf02422d1d84 in kudu's branch refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=d1969e2 ]
KUDU-3277 fix concurrent session flush issue in Java client
This patch fixes the issue reported by KUDU-3277. The bug manifested
itself in rare cases when a session running in the AUTO_FLUSH_BACKGROUND
mode was either being flushed explicitly or implicitly (i.e. upon
closing the session) while the AUTO_FLUSH_BACKGROUND session's logic was
flushing its data buffers concurrently as well.
This patch also adds a reproduction scenario for KUDU-3277. The newly
introduced test scenario was reliably failing before the fix:
* 'java.lang.AssertionError: This Deferred was already called'
messages were encountered in the log multiple times with the stack
exactly as described in KUDU-3277
* some flusher threads were unable to join since KuduSession.flush()
would hang (i.e. would not return)
Change-Id: If6aaccc06abf1a2673620ab7c649f51f91999ad9
Reviewed-on: http://gerrit.cloudera.org:8080/17486
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
> unexpected exception from downstream with kudu client 1.10
> -----------------------------------------------------------
>
> Key: KUDU-3277
> URL: https://issues.apache.org/jira/browse/KUDU-3277
> Project: Kudu
> Issue Type: Bug
> Components: client, java
> Affects Versions: 1.10.0, 1.11.0, 1.12.0, 1.11.1, 1.13.0, 1.14.0
> Environment: flink-connector-kudu
> Reporter: wangkang
> Assignee: Alexey Serbin
> Priority: Major
> Labels: client
>
> flink-connector-kudu:
> {code:java}
> 2021-04-20 00:19:02,451 ERROR org.apache.kudu.client.Connection - [peer 26a2b2540158413ebf18206e5864a770(kudu-tserver:7050)] unexpected exception from downstream on [id: 0x9309d12d, /.....:60166 => ....:7050] java.lang.AssertionError: This Deferred was already called! New result=null, this=Deferred@75271701(state=DONE, result=null, callback=<none>, errback=<none>) at com.stumbleupon.async.Deferred.callback(Deferred.java:979) at org.apache.kudu.client.AsyncKuduSession$Buffer.callbackFlushNotification(AsyncKuduSession.java:858) at org.apache.kudu.client.AsyncKuduSession.queueBuffer(AsyncKuduSession.java:311) at org.apache.kudu.client.AsyncKuduSession.access$900(AsyncKuduSession.java:114) at org.apache.kudu.client.AsyncKuduSession$TabletLookupCB$1.call(AsyncKuduSession.java:406) at org.apache.kudu.client.AsyncKuduSession$TabletLookupCB$1.call(AsyncKuduSession.java:403) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:173) at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142) at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36) at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> {code}
> one of flink task manager node occur with above exception, it didn't cause flink job fail,but cause whole job stopped writting to kudu server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)