You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ForinMH (via GitHub)" <gi...@apache.org> on 2025/07/24 16:12:46 UTC
[I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
ForinMH opened a new issue, #13612:
URL: https://github.com/apache/hudi/issues/13612
**_Tips before filing an issue_**
DoI need to add my custom class to the Hudi source code and then recompile it? Am I right?
**Describe the problem you faced**
Alternatively, what other approaches exist to perform additional operations after committing data within the Hudi framework?
**To Reproduce**
Steps to reproduce the behavior:
The configuration:
"hoodie.write.commit.callback.on", "true"
"hoodie.write.commit.callback.class", "com.example.HoodieWriteStatsCallback"
The error:
org.apache.hudi.exception.HoodieException: Unable to load class
at org.apache.hudi.common.util.ReflectionUtils.lambda$getClass$0(ReflectionUtils.java:55)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1688)
at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:51)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:123)
at org.apache.hudi.callback.util.HoodieCommitCallbackFactory.create(HoodieCommitCallbackFactory.java:34)
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:93)
at org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:992)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:534)
at org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$write$1(HoodieSparkSqlWriter.scala:190)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.adapter.BaseSpark3Adapter.sqlExecutionWithNewExecutionId(BaseSpark3Adapter.scala:105)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:212)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:127)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:170)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
Caused by: java.lang.ClassNotFoundException: com.example.HoodieWriteStatsCallback
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hudi.common.util.ReflectionUtils.lambda$getClass$0(ReflectionUtils.java:53)
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 1.0.0
* Spark version : 3.3.4
* Hive version : 2
* Hadoop version : 3.3.2
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
Posted by "ForinMH (via GitHub)" <gi...@apache.org>.
ForinMH commented on issue #13612:
URL: https://github.com/apache/hudi/issues/13612#issuecomment-3125914975
> Hello [@ForinMH](https://github.com/ForinMH),
>
> You can package your custom classes into a JAR file and include it in your Spark job by using the --jars option with the spark-submit command. This allows Spark to distribute your custom code across the cluster during execution.
>
> Please let me know if you need any further assistance or clarification.
>
> Thank you.
Thank you, Can this custom class use a parameterized constructor? If so, how to provide parameters? And does this class run on the driver or executor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #13612: [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class.
URL: https://github.com/apache/hudi/issues/13612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13612:
URL: https://github.com/apache/hudi/issues/13612#issuecomment-3219064346
Hi @ForinMH
The provided solutions will resolve your issue. We're now closing this ticket, but feel free to reopen it if the problem persists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #13612:
URL: https://github.com/apache/hudi/issues/13612#issuecomment-3118538487
Hello @ForinMH,
You can package your custom classes into a JAR file and include it in your Spark job by using the --jars option with the spark-submit command. This allows Spark to distribute your custom code across the cluster during execution.
Please let me know if you need any further assistance or clarification.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] I want to use a write callback class to calculate write volume, update volume, and perform some other operations, but Hudi cannot load my custom callback class. [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13612:
URL: https://github.com/apache/hudi/issues/13612#issuecomment-3127592097
Hi @ForinMH
The custom Callback classes will be instantiate at runtime by taking input as HoodieWriteConfig parameter by using **HoodieCommitCallbackFactory.create()** [2]. You can refer the following classes for implementation.
1. [HoodieWriteCommitHttpCallback.java](https://github.com/wangxianghu/hudi/blob/b01a062012c9d17efa5ac92637302d0a3f4aaa08/hudi-client/src/main/java/org/apache/hudi/callback/impl/HoodieWriteCommitHttpCallback.java#L40)
2. [HoodieCommitCallbackFactory.java](https://github.com/apache/hudi/blob/7e8c7e0016464b1b404ca402633cfa2f6d737c4e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/callback/util/HoodieCommitCallbackFactory.java#L31C43-L31C49)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org