You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/12/09 08:45:05 UTC
[GitHub] [iotdb] lohcve opened a new issue, #8403: [Bug]
lohcve opened a new issue, #8403:
URL: https://github.com/apache/iotdb/issues/8403
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar.
### Version
1.0.0
### Describe the bug and provide the minimal reproduce step
配置好集群，然后批量写入数据，到最后快完成的时候报错。试了三次都这样，插入的数据量为377万多条。
核心代码：
`List<String> nodeUrls = Arrays.asList("192.168.145.41:6667", "192.168.145.42:6667", "192.168.145.43:6667");
Session session = new Session.Builder().fetchSize(10000).nodeUrls(nodeUrls).build();
session.open(false);`
`Tablet tablet = new Tablet(ROOT_S1_D1, schemaList, 10000);`
奇怪的是new Tablet的最后一个参数设置为1000不报错，但是设置为10000就报错。
报错信息：
`org.apache.iotdb.rpc.StatementExecutionException: 605: org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
at org.apache.iotdb.rpc.RpcUtils.verifySuccess(RpcUtils.java:96)
at org.apache.iotdb.rpc.RpcUtils.verifySuccessWithRedirection(RpcUtils.java:116)
at org.apache.iotdb.session.SessionConnection.insertTablet(SessionConnection.java:607)
at org.apache.iotdb.session.Session.insertTablet(Session.java:2281)
at com.dameng.dmgeo.demo.iotdb.IoTDBDemoApplicationTests.saveData(IoTDBDemoApplicationTests.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:141)
at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)`
### What did you expect to see?
写入正常
### What did you see instead?
写入报错，抛出异常
### Anything else?
没有了
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] OneSizeFitsQuorum commented on issue #8403: [Bug] 集群写入报错RatisRequestFailedException: Ratis request failed
Posted by GitBox <gi...@apache.org>.
OneSizeFitsQuorum commented on issue #8403:
URL: https://github.com/apache/iotdb/issues/8403#issuecomment-1344396513
Currently, our Raft consensus algorithm restricts the size of each write request to be no more than 4M. According to the error message, when you insert with a batchSize of 10000, a write request reaches 6M. You can restart iotdb after changing `data_region_ratis_log_appender_buffer_size_max` in `iotdb-common.properities` to `16777216` which means 16M in confignode configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] lohcve commented on issue #8403: [Bug] 集群写入报错RatisRequestFailedException: Ratis request failed
Posted by GitBox <gi...@apache.org>.
lohcve commented on issue #8403:
URL: https://github.com/apache/iotdb/issues/8403#issuecomment-1344357015
2022-12-09 16:40:37,897 [pool-28-IoTDB-ClientRPC-Processor-7$20221209_084034_04118_3.1.0] WARN o.a.i.d.m.e.e.RegionWriteExecutor$WritePlanNodeExecutionVisitor:246 - Something wrong happened while calling consensus layer's write API.
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
at org.apache.iotdb.consensus.ratis.RatisConsensus.write(RatisConsensus.java:288)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.fireTriggerAndInsert(RegionWriteExecutor.java:112)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.executeDataInsert(RegionWriteExecutor.java:226)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:174)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:128)
at org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode.accept(InsertTabletNode.java:1086)
at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:86)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:234)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
at org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:288)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:206)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:1234)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4078)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4058)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: org.apache.ratis.server.raftlog.RaftLogIOException from Server 3@group-000100000018: Log entry size 6775448 exceeds the max buffer limit of 4194304
at org.apache.ratis.server.raftlog.RaftLogBase.appendImpl(RaftLogBase.java:184)
at org.apache.ratis.server.raftlog.RaftLogBase.lambda$append$2(RaftLogBase.java:161)
at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:69)
at org.apache.ratis.server.raftlog.RaftLogBase.append(RaftLogBase.java:161)
at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:366)
at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:770)
at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:878)
at org.apache.ratis.server.impl.RaftServerImpl.lambda$null$12(RaftServerImpl.java:815)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$13(RaftServerImpl.java:815)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 3 common frames omitted
Caused by: org.apache.ratis.server.raftlog.RaftLogIOException: Log entry size 6775448 exceeds the max buffer limit of 4194304
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:259)
at org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:426)
at org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:412)
at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:383)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.ratis.grpc.client.GrpcClientRpc$1.onNext(GrpcClientRpc.java:151)
at org.apache.ratis.grpc.client.GrpcClientRpc$1.onNext(GrpcClientRpc.java:148)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:474)
at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onMessage(DelayedClientCall.java:452)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:646)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 common frames omitted
2022-12-09 16:40:37,898 [pool-28-IoTDB-ClientRPC-Processor-7$20221209_084034_04118_3.1.0] WARN o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:236 - write locally failed. TSStatus: TSStatus(code:605, message:org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed), message: org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on issue #8403: [Bug] 集群写入报错RatisRequestFailedException: Ratis request failed
Posted by GitBox <gi...@apache.org>.
HTHou commented on issue #8403:
URL: https://github.com/apache/iotdb/issues/8403#issuecomment-1344114392
Attach the error logs of datanode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] lohcve closed issue #8403: [Bug] 集群写入报错RatisRequestFailedException: Ratis request failed
Posted by GitBox <gi...@apache.org>.
lohcve closed issue #8403: [Bug] 集群写入报错RatisRequestFailedException: Ratis request failed
URL: https://github.com/apache/iotdb/issues/8403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org