You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/15 17:04:44 UTC
[GitHub] [hudi] ssomuah opened a new issue #1836: [SUPPORT] org.apache.hudi.exception.HoodieException: Unable to instantiate payload class
ssomuah opened a new issue #1836:
URL: https://github.com/apache/hudi/issues/1836
**Describe the problem you faced**
It appears that I'm unable to instantiate my custom "hoodie.datasource.write.payload.class" I get an exception saying
Caused by: java.lang.NoSuchMethodException: com.myCustompayloadClass.<init>(org.apache.hudi.common.util.Option)
The constructor for my class is
```
class myCustompayloadClass(rec: GenericRecord, oVal: Comparable[_])
extends BaseAvroPayload(rec, oVal) with HoodieRecordPayload[myCustompayloadClass] {
```
**To Reproduce**
Steps to reproduce the behavior:
1.Spark-submit an application that uses a MOR table and custom payload class
**Expected behavior**
Application should be compact MOR table
**Environment Description**
* Hudi version : Master @ 3b9a30528bd6a6369181702303f3384162b04a7f https://github.com/apache/hudi/tree/3b9a30528bd6a6369181702303f3384162b04a7f
* Spark version :2.4.4
* Hive version : N/A
* Hadoop version :2.7.3
* Storage (HDFS/S3/GCS..) : ABFSS
* Running on Docker? (yes/no) :yes
**Additional context**
It's running on Databricks.
We build a fat jar with hudi-spark included and use it as the argument to the spark-submit.
**Stacktrace**
```Add the stacktrace of the error.```
Driver Log
org.apache.hudi.exception.HoodieIOException: IOException when reading log file
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scan(AbstractHoodieLogRecordScanner.java:245)
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:81)
at org.apache.hudi.table.action.compact.HoodieMergeOnReadTableCompactor.compact(HoodieMergeOnReadTableCompactor.java:127)
at org.apache.hudi.table.action.compact.HoodieMergeOnReadTableCompactor.lambda$compact$644ebad7$1(HoodieMergeOnReadTableCompactor.java:98)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1043)
Executor Log
20/07/15 13:11:13 ERROR AbstractHoodieLogRecordScanner: Got exception when reading log file
org.apache.hudi.exception.HoodieException: Unable to instantiate payload class
at org.apache.hudi.common.util.ReflectionUtils.loadPayload(ReflectionUtils.java:78)
at org.apache.hudi.common.util.SpillableMapUtils.convertToHoodieRecordPayload(SpillableMapUtils.java:116)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.processDataBlock(AbstractHoodieLogRecordScanner.java:277)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.processQueuedBlocksForInstant(AbstractHoodieLogRecordScanner.java:306)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scan(AbstractHoodieLogRecordScanner.java:239)
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:81)
at org.apache.hudi.table.action.compact.HoodieMergeOnReadTableCompactor.compact(HoodieMergeOnReadTableCompactor.java:127)
at org.apache.hudi.table.action.compact.HoodieMergeOnReadTableCompactor.lambda$compact$644ebad7$1(HoodieMergeOnReadTableCompactor.java:98)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1043)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1252)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1226)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1226)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1045)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:315)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:353)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:317)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:140)
at org.apache.spark.scheduler.Task.run(Task.scala:113)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:537)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1541)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:543)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: com.myCustompayloadClass.<init>(org.apache.hudi.common.util.Option)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.hudi.common.util.ReflectionUtils.loadPayload(ReflectionUtils.java:76)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ssomuah commented on issue #1836: [SUPPORT] org.apache.hudi.exception.HoodieException: Unable to instantiate payload class
Posted by GitBox <gi...@apache.org>.
ssomuah commented on issue #1836:
URL: https://github.com/apache/hudi/issues/1836#issuecomment-659089308
Thank you Balaji, that was it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1836: [SUPPORT] org.apache.hudi.exception.HoodieException: Unable to instantiate payload class
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1836:
URL: https://github.com/apache/hudi/issues/1836#issuecomment-659046835
@ssomuah : If you look at other concrete implementations of HoodieRecordPayload, there are 2 constructors defined. For example :
```
public OverwriteWithLatestAvroPayload(GenericRecord record, Comparable orderingVal) {
super(record, orderingVal);
}
public OverwriteWithLatestAvroPayload(Option<GenericRecord> record) {
this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
}
```
You would need the same kind of constructors in your custom payload.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ssomuah closed issue #1836: [SUPPORT] org.apache.hudi.exception.HoodieException: Unable to instantiate payload class
Posted by GitBox <gi...@apache.org>.
ssomuah closed issue #1836:
URL: https://github.com/apache/hudi/issues/1836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org