You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/08 13:18:25 UTC

[GitHub] [hudi] sstimmel opened a new issue, #7409: [SUPPORT]

sstimmel opened a new issue, #7409:
URL: https://github.com/apache/hudi/issues/7409

   
   See the Following exception in the logs after running deltastreamer.  I believe this is shown when we have metrics enabled.
   java.lang.NoSuchMethodError: 'int org.apache.hudi.avro.model.HoodieRollbackMetadata.getTotalFilesDeleted()'
   
   
   
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version : 0.12.1
   
   * Spark version : 3.3.1
   
   * Hive version : 
   
   * Hadoop version : 3
   
   * Storage (HDFS/S3/GCS..) :  S3
   
   * Running on Docker? (yes/no) : yes, kubernetes
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   The following hudi jars are in the classpath
   -hudi-aws-bundle.jar	
   - hudi-cli.jar
   -hudi-hive-sync-bundle.jar
   -hudi-spark-bundle.jar
   -hudi-utilities-slim-bundle.jar
   
   hudi-utilities-slim-bundle does show that class.
    10851 Mon Oct 17 22:29:30 UTC 2022 org/apache/hudi/avro/model/HoodieRollbackMetadata.class
   
   **Stacktrace**
   
   ```22/12/07 16:27:35 INFO org.apache.spark.internal.Logging: Disabling executor 1.
   22/12/07 16:27:35 DEBUG org.apache.spark.internal.Logging: Executor 1 on 10.22.164.85 lost, but reason not yet known.
   22/12/07 16:27:35 DEBUG org.apache.spark.internal.Logging: Cleaning up TaskScheduler state for tasks [] on failed executor 1
   22/12/07 16:27:35 ERROR org.apache.spark.internal.Logging: Uncaught exception in thread dispatcher-CoarseGrainedScheduler
   org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
   	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:178)
   	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:150)
   	at org.apache.spark.rpc.netty.NettyRpcEnv.send(NettyRpcEnv.scala:193)
   	at org.apache.spark.rpc.netty.NettyRpcEndpointRef.send(NettyRpcEnv.scala:563)
   	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$reviveOffers$1(CoarseGrainedSchedulerBackend.scala:630)
   	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1484)
   	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.reviveOffers(CoarseGrainedSchedulerBackend.scala:630)
   	at org.apache.spark.scheduler.TaskSchedulerImpl.executorLost(TaskSchedulerImpl.scala:1004)
   	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.disableExecutor(CoarseGrainedSchedulerBackend.scala:482)
   	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.$anonfun$onDisconnected$1(KubernetesClusterSchedulerBackend.scala:328)
   	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.$anonfun$onDisconnected$1$adapted(KubernetesClusterSchedulerBackend.scala:328)
   	at scala.Option.foreach(Option.scala:407)
   	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.onDisconnected(KubernetesClusterSchedulerBackend.scala:328)
   	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:141)
   	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
   	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
   	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
   	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
   	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   22/12/07 16:27:35 WARN org.apache.spark.internal.Logging: Kubernetes client has been closed.
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: MapOutputTrackerMasterEndpoint stopped!
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: MemoryStore cleared
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: BlockManager stopped
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: BlockManagerMaster stopped
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: OutputCommitCoordinator stopped!
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Successfully stopped SparkContext
   Exception in thread "main" java.lang.NoSuchMethodError: 'int org.apache.hudi.avro.model.HoodieRollbackMetadata.getTotalFilesDeleted()'
   	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:774)
   	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1194)
   	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1177)
   	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1165)
   	at org.apache.hudi.client.BaseHoodieWriteClient.lambda$startCommitWithTime$97cdbdca$1(BaseHoodieWriteClient.java:966)
   	at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:153)
   	at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:965)
   	at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:958)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.startCommit(DeltaSync.java:677)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:581)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:336)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:204)
   	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:202)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:571)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
   	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
   	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
   	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
   	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
   	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
   	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
   	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Shutdown hook called
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-7a432e99-1f82-46a4-a555-10bd6090a2ba
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-42983bb3-9835-4bf4-9ea5-c68004077e7c
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-6098afd8-d431-4b91-a81e-3ebea9419630
   22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-f394dce8-567a-461d-b559-c953966adb93
   22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
   22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
   22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #7409:
URL: https://github.com/apache/hudi/issues/7409#issuecomment-1348497185

   @sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim bundle? also can you confirm you're using spark3.3 bundle ? 
   
   For @lokeshj1703 to follow up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] sstimmel commented on issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
sstimmel commented on issue #7409:
URL: https://github.com/apache/hudi/issues/7409#issuecomment-1348896935

   > @sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim bundle? also can you confirm you're using spark3.3 bundle ?
   > 
   > For @lokeshj1703 to follow up.
   
   i tried removing hudi-cli jar from classpath.  However still see the issue occassionally.  I'm not sure how I can remove hudi-hive-sync-bundle, i need that for hive-syncs?, removing that i get different errors.  I am using spark3.3 bundle. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] sstimmel commented on issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
sstimmel commented on issue #7409:
URL: https://github.com/apache/hudi/issues/7409#issuecomment-1349025430

   i removed hudi-sync-bundle and switched over to use hudi-utilities-bundle instead of hudi-utilities-slim-bundle, since that has hive-sync included, but still see the error with that as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua closed issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
yihua closed issue #7409: [SUPPORT]
URL: https://github.com/apache/hudi/issues/7409


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
yihua commented on issue #7409:
URL: https://github.com/apache/hudi/issues/7409#issuecomment-1363374426

   Hi @sstimmel Thanks for the feedback.  I created a ticket to track the improvement: [HUDI-5466](https://issues.apache.org/jira/browse/HUDI-5466).  Closing this issue now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] sstimmel commented on issue #7409: [SUPPORT]

Posted by GitBox <gi...@apache.org>.
sstimmel commented on issue #7409:
URL: https://github.com/apache/hudi/issues/7409#issuecomment-1349681574

   this i believe is resolved now.  Having to build a custom hudi-aws-bundle jar that removes hudi-common from it.  Also needed to add aws-sts to the bundle to run it from Kubernetes (EKS).  Would be nice to not have to do that for all future releases though


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org