You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:30:40 UTC
[I] spark sql query fail on mor table after flink cdc delete records [hudi]
hudi-bot opened a new issue, #15795:
URL: https://github.com/apache/hudi/issues/15795
after flink cdc application delete records for a mor table, spark sql will query fail on the table with below exception:
Serialization trace:
orderingVal (org.apache.hudi.common.model.DeleteRecord)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:104)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:78)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.deserialize(HoodieDeleteBlock.java:106)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.getRecordsToDelete(HoodieDeleteBlock.java:91)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:473)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:343)
... 23 more
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.avro.util.Utf8
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
... 37 more
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5798
- Type: Bug
---
## Comments
15/Feb/23 12:57;lrz;I fix this issue by add a special avro shade jar at spark/jars, and it seems not good to introduce into hudi project ;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] spark sql query fail on mor table after flink cdc delete records [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15795:
URL: https://github.com/apache/hudi/issues/15795#issuecomment-4795476986
This issue should be fixed on master.
The root cause was that Flink wrote delete log blocks via Kryo, which embedded the shaded class name `org.apache.hudi.org.apache.avro.util.Utf8` from the Flink bundle into the bytes; Spark then failed to load that shaded class on read.
Two fixes resolve it:
1. **HUDI-4959 / #7024** — `SerializationUtils` now explicitly registers `Utf8.class` with a custom `AvroUtf8Serializer` that serializes the raw bytes rather than relying on Kryo's default field-based serializer (which embedded the class name).
2. **HUDI-5760 / #9315** (commit `211c315e57f8`) — delete log blocks were upgraded to V3, which uses Avro (`HoodieDeleteRecordList.avsc`) instead of Kryo entirely. See `HoodieDeleteBlock.serializeV3()`. Kryo is only retained as a read-fallback for legacy V1/V2 blocks.
New tables written by current Flink + read by current Spark no longer hit this path. If you can still reproduce on master with a fresh table, please reopen with the Flink/Spark/Hudi versions used. Closing as fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] spark sql query fail on mor table after flink cdc delete records [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15795:
URL: https://github.com/apache/hudi/issues/15795#issuecomment-3630124993
### Linked PR(s)
- https://github.com/apache/hudi/pull/7957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] spark sql query fail on mor table after flink cdc delete records [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15795: spark sql query fail on mor table after flink cdc delete records
URL: https://github.com/apache/hudi/issues/15795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org