You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/01 03:59:28 UTC
[GitHub] [hudi] garyli1019 opened a new issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
garyli1019 opened a new issue #1890:
URL: https://github.com/apache/hudi/issues/1890
I am testing the MOR Spark data source support #1848 . The entire dataset was 100+GB, and I upsert 3GB duplicate data into 30 partitions. Each upserted partition has a 100MB parquet file and 1GB+ log file with exact same records. The merging logic works fine but for 4 partitions, it's failed due to a kryo exception when `HoodieMergedLogRecordScanner.getRecords.get(key)". It always gets stuck on a single record. I am guessing there is a corrupted record in the log file. Have you guys seen this before?
@vinothchandar @n3nash
Caused by: com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$45/62103784
Serialization trace:
orderingVal (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload)
data (org.apache.hudi.common.model.HoodieRecord)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:107)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:81)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:217)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:211)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:207)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.get(ExternalSpillableMap.java:168)
at org.apache.hudi.common.util.collection.ExternalSpillableMap.get(ExternalSpillableMap.java:55)
at org.apache.hudi.HoodieMergeOnReadRDD$$anon$1.hasNext(HoodieMergeOnReadRDD.scala:128)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithoutKey_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:624)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$45/62103784
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
... 31 more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-666851910
@bvaradar Yes I can reproduce this with 0.5.3 during the compaction.
```
com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
Serialization trace:
orderingVal (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload)
data (org.apache.hudi.common.model.HoodieRecord)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:107)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:81)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:217)
at org.apache.hudi.common.util.collection.LazyFileIterable$LazyFileIterator.next(LazyFileIterable.java:92)
at org.apache.hudi.common.util.collection.ExternalSpillableMap$IteratorWrapper.next(ExternalSpillableMap.java:285)
at org.apache.hudi.io.HoodieMergeHandle.close(HoodieMergeHandle.java:317)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:209)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:190)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.compact(HoodieMergeOnReadTableCompactor.java:139)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.lambda$compact$644ebad7$1(HoodieMergeOnReadTableCompactor.java:98)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1193)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1102)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:893)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
... 42 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 edited a comment on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 edited a comment on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667465216
This issue happened to me again. Now the cause could be narrowed down.
When the log file was larger than `HoodieStorageConfig.LOGFILE_SIZE_MAX_BYTES`(1GB in default), the log file will be split into two files and the total size of the two log files is larger than 2GB. When loading these two splits, this issue happened.
~~My guess was the serializer has been reset after loading the first file~~ Created a ticket to track this https://issues.apache.org/jira/browse/HUDI-1141
EDIT: Looks like this could be an integer overflow issue. https://github.com/apache/hudi/blame/master/hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java#L354
`Integer.MAXVALUE` is ~2GB. The file size fields in the relevant classes are all `Integer`.
In my test, some log groups are larger than 2GB, so the smaller log file group is fine but large ones were failing.
@bvaradar What do you think? Should we fix this or we should just avoid having such a large log?
This kind of large log file is unusual because I was just stress-testing the merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667062787
@garyli1019 : I am just guessing here -> Could this be due to type of orderingVal ? Is that (de)serializable with Kryo? Can you try to see by writing a quick test if you can serialize and deserialize orderingVal correctly ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zherenyu831 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
zherenyu831 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667103547
We also having it now, the reason is we used custom payload while upsert data, but when we bulk insert at very beginning, we used `org.apache.hudi.common.model.OverwriteWithLatestAvroPayload`, and in hoodie.properties file, the payload class is `org.apache.hudi.common.model.OverwriteWithLatestAvroPayload`, if you are facing same problem as us, just change the payload class in hoodie.properties, it fix the problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-666776616
@garyli1019 : QQ : Are you able to repro this in 0.5.3 (through hive) ? I will take a look tonight/tomorrow morning to get back. @n3nash @satishkotha : Just checking with you if this rings any bell for you ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-685859696
Closing this as it is tracked in https://issues.apache.org/jira/browse/HUDI-1205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 edited a comment on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 edited a comment on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-666851910
@bvaradar Yes I can reproduce this with 0.5.3 during the compaction.
EDIT: Also only in a few partitions
```
com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
Serialization trace:
orderingVal (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload)
data (org.apache.hudi.common.model.HoodieRecord)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:107)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:81)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:217)
at org.apache.hudi.common.util.collection.LazyFileIterable$LazyFileIterator.next(LazyFileIterable.java:92)
at org.apache.hudi.common.util.collection.ExternalSpillableMap$IteratorWrapper.next(ExternalSpillableMap.java:285)
at org.apache.hudi.io.HoodieMergeHandle.close(HoodieMergeHandle.java:317)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:209)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:190)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.compact(HoodieMergeOnReadTableCompactor.java:139)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.lambda$compact$644ebad7$1(HoodieMergeOnReadTableCompactor.java:98)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1193)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1102)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:893)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
... 42 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667465216
This issue happened to me again. Now the cause could be narrowed down.
When the log file was larger than `HoodieStorageConfig.LOGFILE_SIZE_MAX_BYTES`(1GB in default), the log file will be split into two files. When loading these two splits, this issue happened.
My guess was the serializer has been reset after loading the first file. Created a ticket to track this https://issues.apache.org/jira/browse/HUDI-1141
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-668207368
@bvaradar I tried `HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_MERGE_PROP=0.01` and `HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_COMPACTION_PROP=0.01`, but this issue still persist when the log file group is larger than 2GB.
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java#L353
```
// Size (numberOfBytes) of the value written to disk
private Integer sizeOfValue;
```
The max value of Integer to represent byte is 2GB, so I believe this issue is related to this.
For the log file group that was 1.8GB, I don't see this issue anymore. Tried with both low and high memory fraction.
Would you try in your environment to have a log file group larger than 2GB and see if you can reproduce?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #1890:
URL: https://github.com/apache/hudi/issues/1890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 closed issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 closed issue #1890:
URL: https://github.com/apache/hudi/issues/1890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 edited a comment on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 edited a comment on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667402304
~~Solved. I guess it's because I didn't drop the hudi meta columns when reading from a hudi COW source and write to another folder as MOR. The existing hudi meta columns might cause some issues on the schema side maybe.
When dropping the hudi meta columns before writing, the problem was solved. Both compaction and MOR data source reader is working fine.~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zherenyu831 edited a comment on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
zherenyu831 edited a comment on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667103547
We also having it now, the reason is we using custom payload while upsert data, but when we bulk insert at very beginning, we used `org.apache.hudi.common.model.OverwriteWithLatestAvroPayload`, and in hoodie.properties file, the payload class is `org.apache.hudi.common.model.OverwriteWithLatestAvroPayload`, if you are facing same problem as us, just change the payload class in hoodie.properties, it fix the problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667402304
Solved. I guess it's because I didn't drop the hudi meta columns when reading from a hudi COW source and write to another folder as MOR. The existing hudi meta columns might cause some issues on the schema side maybe.
When dropping the hudi meta columns before writing, the problem was solved. Both compaction and MOR data source reader is working fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-668072392
@garyli1019 : From the stack trace, it looks like kryo is throwing exception when reading records from diskbased map. Can you try a simple test where you keep the HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_MERGE_PROP and HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_COMPACTION_PROP to be very low such that records will get serialized to disk during merging/compaction and try to reproduce the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-666765346
Compaction failed with the same error. This is very strange cause most of the log files are just loading fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 edited a comment on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 edited a comment on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-666851910
@bvaradar Yes I can reproduce this with 0.5.3 during the compaction.
EDIT: Also only in a few partitions. Found a related issue #1823
```
com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
Serialization trace:
orderingVal (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload)
data (org.apache.hudi.common.model.HoodieRecord)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:107)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:81)
at org.apache.hudi.common.util.collection.DiskBasedMap.get(DiskBasedMap.java:217)
at org.apache.hudi.common.util.collection.LazyFileIterable$LazyFileIterator.next(LazyFileIterable.java:92)
at org.apache.hudi.common.util.collection.ExternalSpillableMap$IteratorWrapper.next(ExternalSpillableMap.java:285)
at org.apache.hudi.io.HoodieMergeHandle.close(HoodieMergeHandle.java:317)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdateInternal(HoodieCopyOnWriteTable.java:209)
at org.apache.hudi.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:190)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.compact(HoodieMergeOnReadTableCompactor.java:139)
at org.apache.hudi.table.compact.HoodieMergeOnReadTableCompactor.lambda$compact$644ebad7$1(HoodieMergeOnReadTableCompactor.java:98)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1193)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1102)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1167)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:893)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.model.OverwriteWithLatestAvroPayload$$Lambda$101/1933964683
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
... 42 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #1890: [SUPPORT] Failed to get record from HoodieMergedLogRecordScanner
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #1890:
URL: https://github.com/apache/hudi/issues/1890#issuecomment-667254308
@zherenyu831 Thanks for sharing your experience! In my `hoodie.properties` file the payload was `org.apache.hudi.common.model.OverwriteWithLatestAvroPayload`. The strange part is the compaction was just partially failed and most of the log files are ser/deser fine.
@bvaradar The orderingVal I set up for upsert is a `String` type. Not sure if `orderingVal (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload)` mean Kryo is trying to deser `orderingVal` as a OverwriteWithLatestAvroPayload type. I will dig into this deeper today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org