[GitHub] [hudi] yihua commented on pull request #6079: [HUDI-3287] Remove hudi-spark dependencies from hudi-kafka-connect-bundle

[GitBox <gi...@apache.org>]

yihua commented on PR #6079:
URL: https://github.com/apache/hudi/pull/6079#issuecomment-1250015218

   I hit this issue while testing this change with Kafka Connect sink connector for Hudi:
   ```
   [2022-09-16 23:44:48,156] ERROR [hudi-sink|task-3] WorkerSinkTask{id=hudi-sink-3} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:193)
   java.lang.NoClassDefFoundError: org/apache/hudi/keygen/CustomKeyGenerator
   	at org.apache.hudi.connect.utils.KafkaConnectUtils.getPartitionColumns(KafkaConnectUtils.java:189)
   	at org.apache.hudi.connect.writers.KafkaConnectTransactionServices.<init>(KafkaConnectTransactionServices.java:90)
   	at org.apache.hudi.connect.transaction.ConnectTransactionCoordinator.<init>(ConnectTransactionCoordinator.java:88)
   	at org.apache.hudi.connect.HoodieSinkTask.bootstrap(HoodieSinkTask.java:191)
   	at org.apache.hudi.connect.HoodieSinkTask.open(HoodieSinkTask.java:151)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.openPartitions(WorkerSinkTask.java:635)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.access$1000(WorkerSinkTask.java:71)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask$HandleRebalance.onPartitionsAssigned(WorkerSinkTask.java:700)
   	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsAssigned(ConsumerCoordinator.java:293)
   	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:430)
   	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:450)
   	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:366)
   	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:508)
   	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262)
   	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
   	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:452)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:324)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
   	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
   	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
   	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241)
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   ```
   `KafkaConnectUtils` in `hudi-kafka-connect` uses `CustomKeyGenerator` from `hudi-spark-client module`.  @rmahindra123 looks like we cannot get rid of hudi-spark dependencies here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org