You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/16 05:40:17 UTC
[GitHub] [incubator-seatunnel] hailin0 opened a new issue, #2754: [Bug] [Connector-v2-kafka] Failed to running in streaming mode
hailin0 opened a new issue, #2754:
URL: https://github.com/apache/incubator-seatunnel/issues/2754
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
issuss:
- java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
- Duplicate consume message when commit_on_checkpoint ennable
### SeaTunnel Version
dev branch
### SeaTunnel Config
```conf
env {
execution.parallelism = 2
job.mode = "STREAMING"
}
```
### Running Command
```shell
run as example module
```
### Error Exception
```log
22/09/15 17:49:04 WARN Task: Source: SeaTunnel KafkaSource -> Sink Writer: Unnamed (1/1)#0 (2e1f52d0a07ae1993e29025337346e81) switched from RUNNING to FAILED with failure cause: java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2490)
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2474)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1496)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1400)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1357)
at org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceReader.notifyCheckpointComplete(KafkaSourceReader.java:144)
at org.apache.seatunnel.translation.source.ParallelSource.notifyCheckpointComplete(ParallelSource.java:202)
at org.apache.seatunnel.translation.flink.source.BaseSeaTunnelSourceFunction.notifyCheckpointComplete(BaseSeaTunnelSourceFunction.java:116)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:130)
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.lang.RuntimeException: java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
at org.apache.seatunnel.translation.flink.source.BaseSeaTunnelSourceFunction.cancel(BaseSeaTunnelSourceFunction.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:154)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:210)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:191)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:656)
... 4 more
Caused by: java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2490)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:2373)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:2351)
at org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceReader.close(KafkaSourceReader.java:85)
at org.apache.seatunnel.translation.source.ParallelSource.close(ParallelSource.java:149)
at org.apache.seatunnel.translation.flink.source.BaseSeaTunnelSourceFunction.cancel(BaseSeaTunnelSourceFunction.java:107)
... 8 more
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed issue #2754: [Bug] [Connector-v2-kafka] Failed to running in streaming mode
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed issue #2754: [Bug] [Connector-v2-kafka] Failed to running in streaming mode
URL: https://github.com/apache/incubator-seatunnel/issues/2754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #2754: [Bug] [Connector-v2-kafka] Failed to running in streaming mode
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #2754:
URL: https://github.com/apache/incubator-seatunnel/issues/2754#issuecomment-1248940380
@Hisoka-X
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org