You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "刘珍 (Jira)" <ji...@apache.org> on 2022/10/24 03:57:00 UTC
[jira] [Created] (IOTDB-4727) Ratis request failed ： Cannot cast java.lang.NullPointerException to java.io.IOException
刘珍 created IOTDB-4727:
-------------------------
Summary: Ratis request failed ： Cannot cast java.lang.NullPointerException to java.io.IOException
Key: IOTDB-4727
URL: https://issues.apache.org/jira/browse/IOTDB-4727
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Song Ziyang
Attachments: more_ts.conf
master_1023_2fea011
schema region : ratis
data region : multiLeader
3rep , 3C3D
ip66 /data/liuzhen_test/master_1023_2fea011/logs datanode :
2022-10-24 11:21:39,969 [pool-22-IoTDB-DataNodeInternalRPC-Processor-8] ERROR o.a.i.d.m.e.e.RegionWriteExecutor$WritePlanNodeExecutionVisitor:131 - {color:#DE350B}*Something wrong happened while calling consensus layer's write API.*{color}
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
at org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:262)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.executePlanNodeInConsensusLayer(RegionWriteExecutor.java:146)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitPlan(RegionWriteExecutor.java:123)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitPlan(RegionWriteExecutor.java:113)
at org.apache.iotdb.db.mpp.plan.planner.plan.node.PlanVisitor.visitCreateMultiTimeSeries(PlanVisitor.java:214)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:368)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:113)
at org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.write.CreateMultiTimeSeriesNode.accept(CreateMultiTimeSeriesNode.java:141)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:79)
at org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.sendPlanNode(DataNodeInternalRPCServiceImpl.java:227)
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3221)
at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3201)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-B007979C3BB9->5@group-000200000007, cid=16, seq=0, RW, org.apache.iotdb.consensus.ratis.RequestMessage@61a2e19e for 2 attempts with org.apache.iotdb.consensus.ratis.RatisClient$RatisRetryPolicy@25d595d4
at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
at org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:119)
at org.apache.ratis.client.impl.BlockingImpl.send(BlockingImpl.java:85)
at org.apache.ratis.client.impl.BlockingImpl.send(BlockingImpl.java:59)
at org.apache.iotdb.consensus.ratis.RatisConsensus.lambda$writeRemotelyWithRetry$2(RatisConsensus.java:198)
at org.apache.iotdb.consensus.ratis.RatisConsensus.writeWithRetry(RatisConsensus.java:176)
at org.apache.iotdb.consensus.ratis.RatisConsensus.writeRemotelyWithRetry(RatisConsensus.java:198)
at org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:256)
... 17 common frames omitted
Caused by: java.io.IOException: java.lang.NullPointerException
at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:56)
at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:121)
at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:91)
at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:152)
at org.apache.ratis.grpc.client.GrpcClientRpc$1.onError(GrpcClientRpc.java:156)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:468)
at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:432)
at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:465)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 common frames omitted
Caused by: java.lang.NullPointerException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:250)
at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:255)
at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:120)
... 19 common frames omitted
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL
at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:535)
... 16 common frames omitted
Suppressed: java.lang.IllegalStateException: Failed to cast to class java.io.IOException, object=java.lang.NullPointerException
at org.apache.ratis.proto.RaftProtos$ClientMessageEntryProto$Builder.setContent(RaftProtos.java:25672)
at org.apache.ratis.client.impl.ClientProtoUtils.toClientMessageEntryProtoBuilder(ClientProtoUtils.java:493)
at org.apache.ratis.client.impl.ClientProtoUtils.toClientMessageEntryProtoBuilder(ClientProtoUtils.java:497)
at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReplyProto(ClientProtoUtils.java:254)
at org.apache.ratis.grpc.server.GrpcClientProtocolService$UnorderedRequestStreamObserver.lambda$processClientRequest$0(GrpcClientProtocolService.java:286)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.ratis.util.IOUtils.readObject(IOUtils.java:237)
at org.apache.ratis.util.IOUtils.bytes2Object(IOUtils.java:214)
at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:109)
at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:91)
at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:152)
at org.apache.ratis.grpc.client.GrpcClientRpc$1.onError(GrpcClientRpc.java:156)
... 16 common frames omitted
Caused by: java.lang.ClassCastException: Cannot cast java.lang.NullPointerException to java.io.IOException
at java.lang.Class.cast(Class.java:3369)
at org.apache.ratis.util.IOUtils.readObject(IOUtils.java:234)
... 21 common frames omitted
The test environment
1. 192.168.10.62/66/68 72cpu 256GB
benchmark is located on ip64 ： /data/liuzhen_test/weektest/benchmark_tool
ConfigNode ：
MAX_HEAP_SIZE="8G"
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=3
data_replication_factor=3
DataNode ：
MAX_HEAP_SIZE="192G"
MAX_DIRECT_MEMORY_SIZE="32G"
query_timeout_threshold=36000000
2. benchmark configuration
see attachment
3. ip66 datanode logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)