You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "jiangzzwy (via GitHub)" <gi...@apache.org> on 2023/04/17 09:41:21 UTC
[GitHub] [incubator-seatunnel] jiangzzwy opened a new issue, #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
jiangzzwy opened a new issue, #4595:
URL: https://github.com/apache/incubator-seatunnel/issues/4595
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
在测试Seatunnel故障容错能力的时候，代码如下：
`
env {
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 2000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
Kafka {
result_table_name = "kafka_name"
schema = {
fields {
sex = "boolean"
count = "int"
}
}
start_mode = "latest"
topic = "t_count"
format = JSON
bootstrap.servers = "localhost:9092"
}
}
sink {
Console {
}
}
`
输入数据如下：
{"sex":true,"count":1}
{"sex":true,"count":1}
..
aaaa- 脏数据
报错了，服务直接退出了，因为实时同步服务一旦出现脏数据是不期望推出或者服务挂掉的
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 2000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
Kafka {
result_table_name = "kafka_name"
schema = {
fields {
sex = "boolean"
count = "int"
}
}
start_mode = "latest"
topic = "t_count"
format = JSON
bootstrap.servers = "localhost:9092"
}
}
sink {
Console {
}
}
```
### Running Command
```shell
% ./bin/seatunnel.sh --config ./config/kafka.stream.template -e local
```
### Error Exception
```log
2023-04-17 17:36:17,474 ERROR org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation - [localhost]:5801 [seatunnel-611127] [5.1] ErrorCode:[KAFKA-04], ErrorDescription:[Add a split back to the split enumerator failed,it will only happen when a SourceReader failed] - java.lang.NullPointerException
at org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.listOffsets(KafkaSourceSplitEnumerator.java:327)
at org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.convertToNextSplit(KafkaSourceSplitEnumerator.java:191)
at org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.addSplitsBack(KafkaSourceSplitEnumerator.java:182)
at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.addSplitsBack(SourceSplitEnumeratorTask.java:186)
at org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation.lambda$run$1(RestoredSplitOperation.java:98)
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation.run(RestoredSplitOperation.java:82)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61)
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMemberNode(NodeEngineUtil.java:51)
at org.apache.seatunnel.engine.server.execution.TaskExecutionContext.sendToMember(TaskExecutionContext.java:47)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.restoreState(SourceFlowLifeCycle.java:253)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$14(SeaTunnelTask.java:383)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:380)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:96)
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61)
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMemberNode(NodeEngineUtil.java:51)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.sendOperationToMemberNode(CheckpointManager.java:272)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:249)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
jiangzhongzhou@ZBMac-VPFV0G77T apache-seatunnel % java -version
java version "1.8.0_311"
Java(TM) SE Runtime Environment (build 1.8.0_311-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.311-b11, mixed mode)
### Screenshots
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
URL: https://github.com/apache/seatunnel/issues/4595
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4595:
URL: https://github.com/apache/incubator-seatunnel/issues/4595#issuecomment-1552244635
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on issue #4595:
URL: https://github.com/apache/incubator-seatunnel/issues/4595#issuecomment-1511148966
Currently, SeaTunnel does not support handling dirty data, but we are currently designing it, you can get more information from https://github.com/apache/incubator-seatunnel/issues/4587.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4595: [Bug] [seatunnel-engine] RestoredSplitOperation NullPointerException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4595:
URL: https://github.com/apache/seatunnel/issues/4595#issuecomment-1641205871
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org