You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/30 08:53:42 UTC
[GitHub] [hudi] CaesarWangX opened a new issue, #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
CaesarWangX opened a new issue, #6543:
URL: https://github.com/apache/hudi/issues/6543
``Hi team,
I encountered this problem when using hudi0.11:
When I use the my custom MetricsReporter, after setting "hoodie.metrics.on=true" and "hoodie.metrics.reporter.class=MyMetricsReportor", the custom MetricsReporter cannot always be loaded. The error is java.lang.ClassnotFoundException: com.test.MyMetricsReportor, which indicates that this class cannot be found.
However, and hudi0.10 are normal.
Questions :
1. Are there any different configurations in hudi0.11
2. What shall I do?
Here is the trace log
`Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: Unable to load class
at org.apache.hudi.metrics.Metrics.init(Metrics.java:98)
at org.apache.hudi.metrics.HoodieMetrics.<init>(HoodieMetrics.java:61)
at org.apache.hudi.client.BaseHoodieWriteClient.<init>(BaseHoodieWriteClient.java:176)
at org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:95)
at org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:79)
at org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:200)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$11(HoodieSparkSqlWriter.scala:286)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:285)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:165)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:115)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:112)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:108)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:519)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:83)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:519)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:495)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:108)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:95)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:136)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at com.xxx.components.Hudi.write(HudiSink.scala:32)
at com.xxx.components.DAG.$anonfun$doStart$2(DAG.scala:40)
at com.xxx.components.DAG.$anonfun$doStart$2$adapted(DAG.scala:38)
at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:600)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:598)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:375)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:373)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:69)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:598)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:228)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:375)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:373)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:69)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:193)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:187)
at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:303)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:286)
... 1 more
Caused by: org.apache.hudi.exception.HoodieException: Unable to load class
at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:57)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89)
at org.apache.hudi.metrics.MetricsReporterFactory.createReporter(MetricsReporterFactory.java:48)
at org.apache.hudi.metrics.Metrics.<init>(Metrics.java:50)
at org.apache.hudi.metrics.Metrics.init(Metrics.java:96)
... 78 more
Caused by: java.lang.ClassNotFoundException: com.xxx.MyMetricsRepoter
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:54)
... 82 more
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] easonwood commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
easonwood commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236347008
@Zouxxyy
I try your method in https://github.com/apache/hudi/issues/6563.
The classloader result looks correct
[arthas@14524]$ classloader -t
+-BootstrapClassLoader
+-sun.misc.Launcher$ExtClassLoader@7f77e91b
+-com.taobao.arthas.agent.ArthasClassloader@67e8d194
+-sun.misc.Launcher$AppClassLoader@3fb4f649
+-org.apache.spark.util.MutableURLClassLoader@6f44a157
[arthas@14524]$ classloader -c 67e8d194 -load us.zoom.op.utils.OrderlyMergeRecordPayload
load class error, class: us.zoom.op.utils.OrderlyMergeRecordPayload, error: java.lang.ClassNotFoundException: us.zoom.op.utils.OrderlyMergeRecordPayload
[arthas@14524]$ classloader -c 3fb4f649 -load us.zoom.op.utils.OrderlyMergeRecordPayload
load class success.
class-info us.zoom.op.utils.OrderlyMergeRecordPayload
name us.zoom.op.utils.OrderlyMergeRecordPayload
super-class +-org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+-org.apache.hudi.common.model.BaseAvroPayload
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@3fb4f649
+-sun.misc.Launcher$ExtClassLoader@7f77e91b
classLoaderHash 3fb4f649
but my job result is still error
Caused by: java.lang.ClassNotFoundException: us.zoom.op.utils.OrderlyMergeRecordPayload
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:54)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236348594
> customer.jar
where do you add your customer jar? You can try putting it where your hudi-spark-bundle.jar is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236558188
Hi @Zouxxyy
Thanks for your suggestion.
I tried your method, but I still got the same error
spark-submit \
--master yarn \
--deploy-mode cluster \
--name xxxxxxxx \
--queue clustering \
--class com.test.MainClass \
--files test.conf \
--conf spark.driver.extraClassPath=my-jar-1.0.0-SNAPSHOT-jar-with-dependencies.jar \
--jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/spark/external/lib/spark-avro.jar,/usr/lib/spark/external/lib/spark-sql-kafka-0-10.jar,/usr/lib/spark/external/lib/spark-streaming-kafka-0-10-assembly.jar,/usr/lib/spark/external/lib/spark-token-provider-kafka-0-10.jar \
my-jar-1.0.0-SNAPSHOT-jar-with-dependencies.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1235084941
We found that the customized payload could not be loaded too. ClassNotFound
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] easonwood commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
easonwood commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236362534
@Zouxxyy
ReflectionUtils can be found in AppClassLoader.
spark-submit \
--master yarn \
--deploy-mode cluster \
--jars /usr/lib/spark/jars/hudi-spark-bundle.jar,/usr/lib/hive/lib/*.jar,/usr/lib/hudi/cli/lib/*.jar,/usr/lib/hudi/*.jar\
--conf "spark.driver.extraClassPath=custom.jar" \
custom.jar
my pom like this:
hudi-spark3-bundle_2.12: 0.11.0 [provided]
hudi-spark_2.12: 0.11.0 [provided]
hudi-utilities-bundle_2.12: 0.11.0 [provided]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1233684620
Hi @Zouxxyy ,
Yes, I know the change.
I also used CustomizableMetricsReporter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] easonwood commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
easonwood commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1237849400
I found that in previous version, ReflectionUtils is loaded in MutableURLClassLoader, together with user-defined class.
But 0.11 it's loaded in AppClassLoader.
Why it changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1231369252
trace log
[trace.txt](https://github.com/apache/hudi/files/9451349/trace.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1237958773
Close this issue according to https://github.com/apache/hudi/pull/2753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1233224433
What I know is that in 0.11, the user-defined MetricsReporte base class changed from `AbstractUserDefinedMetricsReporter `to `CustomizableMetricsReporter`, have you adjusted it accordingly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1233770517
Hi @codope .
I am sure that the custom reporter exists in my jar package, because I use the same jar in hudi0.10 and hudi0.9, and it worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236348644
> @Zouxxyy I try your method in #6563. The classloader result looks correct, AppClassLoader load the special class
>
> [arthas@14524]$ classloader -t +-BootstrapClassLoader +-sun.misc.Launcher$ExtClassLoader@7f77e91b +-com.taobao.arthas.agent.ArthasClassloader@67e8d194 +-sun.misc.Launcher$AppClassLoader@3fb4f649 +-org.apache.spark.util.MutableURLClassLoader@6f44a157
>
> [arthas@14524]$ classloader -c 7f77e91b -load us.zoom.op.utils.OrderlyMergeRecordPayload load class error, class: us.zoom.op.utils.OrderlyMergeRecordPayload, error: java.lang.ClassNotFoundException: us.zoom.op.utils.OrderlyMergeRecordPayload
>
> [arthas@14524]$ classloader -c 3fb4f649 -load us.zoom.op.utils.OrderlyMergeRecordPayload load class success. class-info us.zoom.op.utils.OrderlyMergeRecordPayload name us.zoom.op.utils.OrderlyMergeRecordPayload super-class +-org.apache.hudi.common.model.OverwriteWithLatestAvroPayload +-org.apache.hudi.common.model.BaseAvroPayload +-java.lang.Object class-loader +-sun.misc.Launcher$AppClassLoader@3fb4f649 +-sun.misc.Launcher$ExtClassLoader@7f77e91b classLoaderHash 3fb4f649
>
> [arthas@14524]$ classloader -c 6f44a157 -load us.zoom.op.utils.OrderlyMergeRecordPayload load class success. class-info us.zoom.op.utils.OrderlyMergeRecordPayload super-class +-org.apache.hudi.common.model.OverwriteWithLatestAvroPayload +-org.apache.hudi.common.model.BaseAvroPayload +-java.lang.Object class-loader +-sun.misc.Launcher$AppClassLoader@3fb4f649 +-sun.misc.Launcher$ExtClassLoader@7f77e91b classLoaderHash 3fb4f649
>
> But my job result is still error Caused by: java.lang.ClassNotFoundException: us.zoom.op.utils.OrderlyMergeRecordPayload at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:54)
where do you add your customer jar? You can try putting it where your hudi-spark-bundle.jar is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1233969292
Is there a problem in **ReflectionUtils**? I can see that UrlClassLoader maybe cause this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236082037
Maybe I can answer your question.
First, hudi uses `ReflectionUtils.java` 's `Class.forName(clazzName)` to load class. The classloader it used is determined by `ReflectionUtils`, which is determined by `hudi-spark-bundle.jar`.
Then, spark has its own defined class loaders, `MutableURLClassLoader` and `ParentClassLoader`, which recognize classes defined by sparkconf `--jars` or `spark.jars`
Besides, java's `AppClassLoader` can recognize classes in java classpath, which can be defined by sparkconf `spark.driver.extraClassPath`
According to java's class loading mechanism, `MutableURLClassLoader` can see all classes loaded by `AppClassLoader`, but `AppClassLoader` can not see any class loaded by `MutableURLClassLoader`
If your `hudi-spark-bundle.jar` is s defined by `spark.driver.extraClassPath` and your customer jar is defined by `--jars` or `spark.jars`, `ReflectionUtils` will use `AppClassLoader` to load your `customer jar`, since it is not defined In the java classpath, there will be `java.lang.ClassnotFoundException`. I guess that's what you probably did
**So I think configing your `customer.jar` in `spark.driver.extraClassPath` will solve your problem**
Finally, why you were able to succeed before, I guess because of this patch [HUDI-1750](https://github.com/apache/hudi/pull/2753), which was modified back in 0.10.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CaesarWangX closed issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
CaesarWangX closed issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
URL: https://github.com/apache/hudi/issues/6543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6543: [SUPPORT] Unable to load class of UserDefinedMetricsReporter in hudi0.11
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6543:
URL: https://github.com/apache/hudi/issues/6543#issuecomment-1233762585
@CaesarWangX Can you confirm that the jar with the custom reporter class is in the classpath. It should have worked. We even have a [test that exercises that code path](https://github.com/apache/hudi/blob/release-0.11.0/hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metrics/TestMetricsReporterFactory.java#L58).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org