You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Song Ziyang (Jira)" <ji...@apache.org> on 2022/11/14 11:19:00 UTC
[jira] [Commented] (IOTDB-4727) Ratis request failed ： Cannot cast java.lang.NullPointerException to java.io.IOException
[ https://issues.apache.org/jira/browse/IOTDB-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633704#comment-17633704 ]
Song Ziyang commented on IOTDB-4727:
------------------------------------
The issue is reported to Ratis Community: https://issues.apache.org/jira/browse/RATIS-1744
> Ratis request failed ： Cannot cast java.lang.NullPointerException to java.io.IOException
> ----------------------------------------------------------------------------------------
>
> Key: IOTDB-4727
> URL: https://issues.apache.org/jira/browse/IOTDB-4727
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Song Ziyang
> Priority: Major
> Attachments: more_ts.conf
>
>
> master_1023_2fea011
> schema region : ratis
> data region : multiLeader
> 3rep , 3C3D
> ip66 /data/liuzhen_test/master_1023_2fea011/logs datanode :
> 2022-10-24 11:21:39,969 [pool-22-IoTDB-DataNodeInternalRPC-Processor-8] ERROR o.a.i.d.m.e.e.RegionWriteExecutor$WritePlanNodeExecutionVisitor:131 - {color:#DE350B}*Something wrong happened while calling consensus layer's write API.*{color}
> org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
> at org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:262)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.executePlanNodeInConsensusLayer(RegionWriteExecutor.java:146)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitPlan(RegionWriteExecutor.java:123)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitPlan(RegionWriteExecutor.java:113)
> at org.apache.iotdb.db.mpp.plan.planner.plan.node.PlanVisitor.visitCreateMultiTimeSeries(PlanVisitor.java:214)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:368)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:113)
> at org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.write.CreateMultiTimeSeriesNode.accept(CreateMultiTimeSeriesNode.java:141)
> at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:79)
> at org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.sendPlanNode(DataNodeInternalRPCServiceImpl.java:227)
> at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3221)
> at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3201)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-B007979C3BB9->5@group-000200000007, cid=16, seq=0, RW, org.apache.iotdb.consensus.ratis.RequestMessage@61a2e19e for 2 attempts with org.apache.iotdb.consensus.ratis.RatisClient$RatisRetryPolicy@25d595d4
> at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> at org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:119)
> at org.apache.ratis.client.impl.BlockingImpl.send(BlockingImpl.java:85)
> at org.apache.ratis.client.impl.BlockingImpl.send(BlockingImpl.java:59)
> at org.apache.iotdb.consensus.ratis.RatisConsensus.lambda$writeRemotelyWithRetry$2(RatisConsensus.java:198)
> at org.apache.iotdb.consensus.ratis.RatisConsensus.writeWithRetry(RatisConsensus.java:176)
> at org.apache.iotdb.consensus.ratis.RatisConsensus.writeRemotelyWithRetry(RatisConsensus.java:198)
> at org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:256)
> ... 17 common frames omitted
> Caused by: java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:56)
> at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:121)
> at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:91)
> at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:152)
> at org.apache.ratis.grpc.client.GrpcClientRpc$1.onError(GrpcClientRpc.java:156)
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:468)
> at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:432)
> at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:465)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
> at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> ... 3 common frames omitted
> {color:#DE350B}*Caused by: java.lang.NullPointerException: null*{color}
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:250)
> at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:255)
> at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:120)
> ... 19 common frames omitted
> {color:#DE350B}*Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL*{color}
> at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:535)
> ... 16 common frames omitted
> Suppressed: java.lang.IllegalStateException: Failed to cast to class java.io.IOException, object=java.lang.NullPointerException
> at org.apache.ratis.proto.RaftProtos$ClientMessageEntryProto$Builder.setContent(RaftProtos.java:25672)
> at org.apache.ratis.client.impl.ClientProtoUtils.toClientMessageEntryProtoBuilder(ClientProtoUtils.java:493)
> at org.apache.ratis.client.impl.ClientProtoUtils.toClientMessageEntryProtoBuilder(ClientProtoUtils.java:497)
> at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReplyProto(ClientProtoUtils.java:254)
> at org.apache.ratis.grpc.server.GrpcClientProtocolService$UnorderedRequestStreamObserver.lambda$processClientRequest$0(GrpcClientProtocolService.java:286)
> at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
> at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
> at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.apache.ratis.util.IOUtils.readObject(IOUtils.java:237)
> at org.apache.ratis.util.IOUtils.bytes2Object(IOUtils.java:214)
> at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:109)
> at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:91)
> at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:152)
> at org.apache.ratis.grpc.client.GrpcClientRpc$1.onError(GrpcClientRpc.java:156)
> ... 16 common frames omitted
> {color:#DE350B}* Caused by: java.lang.ClassCastException: Cannot cast java.lang.NullPointerException to java.io.IOException*{color}
> at java.lang.Class.cast(Class.java:3369)
> at org.apache.ratis.util.IOUtils.readObject(IOUtils.java:234)
> ... 21 common frames omitted
> The test environment
> 1. 192.168.10.62/66/68 72cpu 256GB
> benchmark is located on ip64 ： /data/liuzhen_test/weektest/benchmark_tool
> ConfigNode ：
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> DataNode ：
> MAX_HEAP_SIZE="192G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> query_timeout_threshold=36000000
> 2. benchmark configuration
> see attachment
> 3. ip66 datanode logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)