You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "15810563327 (via GitHub)" <gi...@apache.org> on 2024/08/09 02:54:53 UTC

[I] [Bug] [seatunel] kafka source Exception [seatunnel]

15810563327 opened a new issue, #7351:
URL: https://github.com/apache/seatunnel/issues/7351

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   I encountered an error when using seatunl-2.3.3 to consume Kafka。
   
   ### SeaTunnel Version
   
   seatunel-2.3.3
   
   ### SeaTunnel Config
   
   ```conf
   My  configuration
   env {
     execution.parallelism = 2
     job.mode = "BATCH"
     checkpoint.interval = 10000
   }
   
   source {
     Kafka {
       format = json
       topic = "test-dataMaintain-dataMaintain"
       bootstrap.servers = "192.168.xx.xx:9092"
       kafka.config = {
         client.id = client_1
         max.poll.records = 500
         auto.offset.reset = "earliest"
         enable.auto.commit = "false"
         consumer.group = "seatunnel_kafka"
       }
     }
   }
   sink {
      Console {
      }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh --config ./example/fzq.yaml -e local
   ```
   
   
   ### Error Exception
   
   ```log
   Exception log:
   
   Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.kafka.common.TopicPartition to field org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplit.topicPartition of type org.apache.kafka.common.TopicPartition in instance of org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplit
   	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
   	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
   	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2436)
   	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2354)
   	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2212)
   	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1668)
   	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
   	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
   	at org.apache.seatunnel.common.utils.SerializationUtils.deserialize(SerializationUtils.java:74)
   	at org.apache.seatunnel.api.serialization.DefaultSerializer.deserialize(DefaultSerializer.java:41)
   	at org.apache.seatunnel.api.serialization.DefaultSerializer.deserialize(DefaultSerializer.java:25)
   	at org.apache.seatunnel.engine.server.task.operation.source.AssignSplitOperation.lambda$run$0(AssignSplitOperation.java:67)
   	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
   	at org.apache.seatunnel.engine.server.task.operation.source.AssignSplitOperation.run(AssignSplitOperation.java:54)
   	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
   	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
   	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
   	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
   	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
   	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
   	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
   	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
   
   	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:122)
   	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184)
   	... 2 more
   2024-08-09 10:28:44,103 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   jdk1.8
   
   ### Screenshots
   
   ![sea0](https://github.com/user-attachments/assets/c1f75609-ff05-4c41-8c5e-693bef33ebe5)
   ![sea4](https://github.com/user-attachments/assets/27d995bb-e9c3-4708-91ba-932fae5d23f6)
   
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "summerpunch (via GitHub)" <gi...@apache.org>.
summerpunch commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2644452237

   @123akn  我也遇到了同样的问题,请问你解决了吗


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-3203488407

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7351: [Bug] [seatunel] kafka source Exception
URL: https://github.com/apache/seatunnel/issues/7351


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "15810563327 (via GitHub)" <gi...@apache.org>.
15810563327 commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2283014294

   > please resend standard message to a new topic, and try it again.
   
   a new tipic?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "15810563327 (via GitHub)" <gi...@apache.org>.
15810563327 commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2283015120

   > please resend standard message to a new topic, and try it again.
   
   This is an anomaly that occurred in the consumption topic。
   Why should I resend the message to a new topic?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2345022354

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "ahuljh (via GitHub)" <gi...@apache.org>.
ahuljh commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2278925253

   please resend standard message to a new topic, and try it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-3177306348

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunel] kafka source Exception [seatunnel]

Posted by "123akn (via GitHub)" <gi...@apache.org>.
123akn commented on issue #7351:
URL: https://github.com/apache/seatunnel/issues/7351#issuecomment-2642404189

   > ### Search before asking
   > * [x]  I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   > 
   > ### What happened
   > I encountered an error when using seatunl-2.3.3 to consume Kafka。
   > 
   > ### SeaTunnel Version
   > seatunel-2.3.3
   > 
   > ### SeaTunnel Config
   > ```
   > My  configuration
   > env {
   >   execution.parallelism = 2
   >   job.mode = "BATCH"
   >   checkpoint.interval = 10000
   > }
   > 
   > source {
   >   Kafka {
   >     format = json
   >     topic = "test-dataMaintain-dataMaintain"
   >     bootstrap.servers = "192.168.xx.xx:9092"
   >     kafka.config = {
   >       client.id = client_1
   >       max.poll.records = 500
   >       auto.offset.reset = "earliest"
   >       enable.auto.commit = "false"
   >       consumer.group = "seatunnel_kafka"
   >     }
   >   }
   > }
   > sink {
   >    Console {
   >    }
   > }
   > ```
   > 
   > ### Running Command
   > ./bin/seatunnel.sh --config ./example/fzq.yaml -e local
   > ### Error Exception
   > ```
   > Exception log:
   > 
   > Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.kafka.common.TopicPartition to field org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplit.topicPartition of type org.apache.kafka.common.TopicPartition in instance of org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplit
   > 	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
   > 	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
   > 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2436)
   > 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2354)
   > 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2212)
   > 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1668)
   > 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
   > 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
   > 	at org.apache.seatunnel.common.utils.SerializationUtils.deserialize(SerializationUtils.java:74)
   > 	at org.apache.seatunnel.api.serialization.DefaultSerializer.deserialize(DefaultSerializer.java:41)
   > 	at org.apache.seatunnel.api.serialization.DefaultSerializer.deserialize(DefaultSerializer.java:25)
   > 	at org.apache.seatunnel.engine.server.task.operation.source.AssignSplitOperation.lambda$run$0(AssignSplitOperation.java:67)
   > 	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
   > 	at org.apache.seatunnel.engine.server.task.operation.source.AssignSplitOperation.run(AssignSplitOperation.java:54)
   > 	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
   > 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
   > 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
   > 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
   > 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
   > 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
   > 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
   > 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
   > 
   > 	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:122)
   > 	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184)
   > 	... 2 more
   > 2024-08-09 10:28:44,103 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
   > ```
   > 
   > ### Zeta or Flink or Spark Version
   > _No response_
   > 
   > ### Java or Scala Version
   > jdk1.8
   > 
   > ### Screenshots
   > ![sea0](https://github.com/user-attachments/assets/c1f75609-ff05-4c41-8c5e-693bef33ebe5) ![sea4](https://github.com/user-attachments/assets/27d995bb-e9c3-4708-91ba-932fae5d23f6)
   > 
   > ### Are you willing to submit PR?
   > * [x]  Yes I am willing to submit a PR!
   > 
   > ### Code of Conduct
   > * [x]  I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   我也遇到了同样的问题,请问该问题你后面解决了吗,如果解决了,是如何解决的呢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org