You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:49:31 UTC
[I] Need to upgrade parquet after nested fields fixed [hudi]
hudi-bot opened a new issue, #15413:
URL: https://github.com/apache/hudi/issues/15413
parquet-mr fails to read nested array fields.
See following issues for more discussion:
[https://github.com/apache/hudi/issues/5985]
[https://github.com/apache/hudi/issues/5701]
Essentially, Hudi needs to upgrade parquet version after following parquet issues are fixed:
https://issues.apache.org/jira/browse/PARQUET-1254
https://issues.apache.org/jira/browse/PARQUET-1681
https://issues.apache.org/jira/browse/PARQUET-2069
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4798
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-6242
- Fix version(s):
- 1.1.0
---
## Comments
06/May/25 02:45;vhs;Another relevant fix that is required:
https://issues.apache.org/jira/browse/PARQUET-2450
The test below will reproduce it on master - e4d01dd791e6e41617bd4d9f1df22a792315d4c2
{code:java}
test("Repro Issue: Array of Struct with single inner field") {
withTempDir { tmp =>
val tableName = "hudi_type_test_mor"
spark.sql(
s"""
|CREATE TABLE $tableName (
| uuid STRING,
| precombine_field LONG,
| col_double DOUBLE,
| array_struct ARRAY<STRUCT<inner_f3: STRING>>,
| part_col STRING
|) USING hudi
| LOCATION '${tmp.getCanonicalPath}'
| TBLPROPERTIES (
| primaryKey = 'uuid',
| type = 'mor',
| preCombineField = 'precombine_field'
| )
| PARTITIONED BY (part_col)
""".stripMargin)
// directly write to new parquet file
spark.sql(s"set hoodie.parquet.small.file.limit=0")
spark.sql(s"set hoodie.metadata.compact.max.delta.commits=1")
// partition stats index is enabled together with column stats index
spark.sql(s"set hoodie.metadata.index.column.stats.enable=true")
spark.sql(s"set hoodie.metadata.record.index.enable=true")
spark.sql(s"set hoodie.metadata.index.secondary.enable=true")
// Insert row 1 into partition 'A'
spark.sql(
s"""
| INSERT INTO $tableName VALUES (
| 'uuid1', 1000L, 1.1,
| array(struct('asd'), struct('ghj')),
| 'A'
| )
""".stripMargin)
spark.sql(s"CREATE INDEX idx_double ON $tableName (col_double)")
// Generate log files through updates on partition 'A'
// Allow time for async indexer to run so that error will be thrown
spark.sql(s"UPDATE $tableName SET col_double = col_double + 100, precombine_field = precombine_field + 1 WHERE part_col = 'A'")
}
}{code}
Error:
{code:java}
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2393)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2414)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2433)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2458)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
at org.apache.spark.rdd.PairRDDFunctions.$anonfun$countByKey$1(PairRDDFunctions.scala:367)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
at org.apache.spark.rdd.PairRDDFunctions.countByKey(PairRDDFunctions.scala:367)
at org.apache.spark.api.java.JavaPairRDD.countByKey(JavaPairRDD.scala:314)
at org.apache.hudi.data.HoodieJavaPairRDD.countByKey(HoodieJavaPairRDD.java:109)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.buildProfile(BaseSparkCommitActionExecutor.java:204)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:171)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:151)
at org.apache.hudi.table.action.deltacommit.SparkUpsertPreppedDeltaCommitActionExecutor.execute(SparkUpsertPreppedDeltaCommitActionExecutor.java:44)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:126)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:88)
at org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:146)
at org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:64)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1412)
at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:136)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1047)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1105)
at org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:277)
... 96 more
Caused by: org.apache.hudi.exception.HoodieException: unable to read next record from parquet file
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:54)
at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39)
at org.apache.hudi.common.table.log.HoodieFileSliceReader.hasNextInternal(HoodieFileSliceReader.java:73)
at org.apache.hudi.common.table.log.HoodieFileSliceReader.doHasNext(HoodieFileSliceReader.java:99)
at org.apache.hudi.common.util.collection.CachingIterator.hasNext(CachingIterator.java:32)
at org.apache.hudi.metadata.SecondaryIndexRecordGenerationUtils.getRecordKeyToSecondaryKey(SecondaryIndexRecordGenerationUtils.java:183)
at org.apache.hudi.metadata.SecondaryIndexRecordGenerationUtils.lambda$convertWriteStatsToSecondaryIndexRecords$8386a558$1(SecondaryIndexRecordGenerationUtils.java:138)
at org.apache.hudi.data.HoodieJavaRDD.lambda$flatMap$a6598fcb$1(HoodieJavaRDD.java:160)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:223)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1614)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1524)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1588)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1389)
at org.apache.spark.storage.BlockManager.getOrElseUpdateRDDBlock(BlockManager.scala:1343)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:379)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: optional binary inner_f3 (STRING) is not a group
at org.apache.parquet.schema.Type.asGroupType(Type.java:248)
at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:362)
at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:306)
at org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:79)
at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.<init>(AvroRecordConverter.java:617)
at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:567)
at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:371)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:144)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:98)
at org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33)
at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:146)
at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190)
at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:49)
... 39 more{code}
This is fixed in Parquet-Avro / parquet.version {*}1.14.0{*}.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need to upgrade parquet after nested fields fixed [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #15413:
URL: https://github.com/apache/hudi/issues/15413#issuecomment-5343481174
This issue was reviewed as part of the JIRA-migrated backlog triage and is being closed as already fixed.
The ask here was to upgrade Parquet once the nested-field problem was addressed upstream. That upgrade has landed and shipped.
Verified by reading the default `parquet.version` from the root `pom.xml` at five refs rather than relying on a fixVersion:
| ref | default parquet.version |
|---|---|
| release-0.12.0 | 1.10.1 |
| release-1.0.0 | 1.10.1 |
| release-1.1.0 | 1.10.1 |
| **release-1.2.0** | **1.13.1** |
| master | 1.13.1 |
The bump is `9a988c248736` (*"fix: Update default Parquet version to 1.13.1"*, PR #17941). Containment checked explicitly: comparing it against `release-1.2.0` reports `ahead`.
One limit worth stating: I verified the version bump, not the nested-array read behaviour itself - I did not re-run the reproducers from #5985 / #5701 against 1.13.1. If nested array reads still fail on 1.2.0, that would be a new defect against the current Parquet version
rather than this ticket, whose precondition is now met.
Closing as fixed in **1.2.0**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need to upgrade parquet after nested fields fixed [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #15413: Need to upgrade parquet after nested fields fixed
URL: https://github.com/apache/hudi/issues/15413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org