You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@xtable.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2025/12/12 23:54:20 UTC

[D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

GitHub user parisni created a discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Hey,
AFAIK hudi default for `parquet.avro.write-old-list-structure` is true. In the hudi->iceberg limitation it's mentionned:

> Be sure to enable `parquet.avro.write-old-list-structure=false` for proper compatibility with lists when syncing from Hudi to Iceberg.

I did some tests, and i could find any issues with `array<struct>` hudi parquet table, read with both spark and athena.

Can someone be more specific on this issue and the risks ?


Thanks

GitHub link: https://github.com/apache/incubator-xtable/discussions/770

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.

GitHub user the-other-tim-brown added a comment to the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Did you confirm that the file is 2-level? If I remember correctly, different paths in Hudi may produce different list structure depending on the underlying writer used (avro vs spark native). 

GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15262582

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "parisni (via GitHub)" <gi...@apache.org>.

GitHub user parisni edited a comment on the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Hi Tim. I did some experimentations and i confirm iceberg spark reader fails in case the of old style list. Still, iceberg athena reader does not care old or new style list.

I also confirm setting `spark.hadoop.parquet.avro.write-old-list-structure=false` when writing with hudi fixes the iceberg spark reader.

I was curious if a given table can mixe both `parquet.avro.write-old-list-structure` = `false or true`. This is not the case when one use the avro payload merger: the upsert fails to read the parquet files.

Interestingly when using `org.apache.hudi.HoodieSparkRecordMerger`, it does use the new style list by default. Moreover it does not honour the `spark.hadoop.parquet.avro.write-old-list-structure=true` either.

My conclusion are:
- `parquet.avro.write-old-list-structure=false` is mandatory if one use the avro payload merger and they need to rewrite the whole table
- in case one use the org.apache.hudi.HoodieSparkRecordMerger, then it's all right 
- in case one use the `bulk_insert` it always produce new style lists too

@the-other-tim-brown can you confirm ?


GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15479738

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "parisni (via GitHub)" <gi...@apache.org>.

GitHub user parisni added a comment to the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Hi Tim. I did some experimentations and i confirm iceberg spark reader fails in case the of old style list. Still, iceberg athena reader does not care old or new style list.

I also confirm setting `spark.hadoop.parquet.avro.write-old-list-structure=false` when writing with hudi fixes the iceberg spark reader.

I was curious if a given table can mixe both `parquet.avro.write-old-list-structure` = `false or true`. This is not the case when one use the avro payload merger: the upsert fails to read the parquet files.

Interestingly when using `org.apache.hudi.HoodieSparkRecordMerger`, it does use the new style list by default. Moreover it does not honour the `spark.hadoop.parquet.avro.write-old-list-structure=false` either.

My conclusion are:
- `parquet.avro.write-old-list-structure=false` is mandatory if one use the avro payload merger and they need to rewrite the whole table
- in case one use the org.apache.hudi.HoodieSparkRecordMerger, then it's all right 
- in case one use the `bulk_insert` it always produce new style lists

@the-other-tim-brown can you confirm ?


GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15479738

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "parisni (via GitHub)" <gi...@apache.org>.

GitHub user parisni edited a comment on the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Hi Tim. I did some experimentations and i confirm iceberg spark reader fails in case the of old style list. Still, iceberg athena reader does not care old or new style list.

I also confirm setting `spark.hadoop.parquet.avro.write-old-list-structure=false` when writing with hudi fixes the iceberg spark reader.

I was curious if a given table can mixe both `parquet.avro.write-old-list-structure` = `false or true`. This is not the case when one use the avro payload merger: the upsert fails to read the parquet files.

Interestingly when using `org.apache.hudi.HoodieSparkRecordMerger`, it does use the new style list by default. Moreover it does not honour the `spark.hadoop.parquet.avro.write-old-list-structure=true` either.

My conclusion are:
- `parquet.avro.write-old-list-structure=false` is mandatory if one use the avro payload merger and they need to rewrite the whole table
- in case one use the org.apache.hudi.HoodieSparkRecordMerger, then it's all right 
- in case one use the `bulk_insert` it always produce new style lists

@the-other-tim-brown can you confirm ?


GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15479738

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.

GitHub user the-other-tim-brown added a comment to the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

A given table could mix the list types but it is not recommended.

The avro config is only applicable to avro writers, that is why the spark writer does not respect it. 

Your conclusions match my understanding.

GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15542143

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "parisni (via GitHub)" <gi...@apache.org>.

GitHub user parisni edited a discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

Hey,
AFAIK hudi default for `parquet.avro.write-old-list-structure` is true. In the hudi->iceberg limitation it's mentionned:

> Be sure to enable `parquet.avro.write-old-list-structure=false` for proper compatibility with lists when syncing from Hudi to Iceberg.

I did some tests, and i could't find any issues with `array<struct>` hudi parquet table, read with both spark and athena.

Can someone be more specific on this issue and the risks ?


Thanks

GitHub link: https://github.com/apache/incubator-xtable/discussions/770

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "parisni (via GitHub)" <gi...@apache.org>.

GitHub user parisni added a comment to the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

> We saw that this meant that some engines were unable to read the data at the time.

Thanks . both spark and athena are fine with 2 lvl. Didn't tryied other engine.

GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15262572

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.

GitHub user the-other-tim-brown added a comment to the discussion: what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

@parisni the Iceberg [spec](https://iceberg.apache.org/spec/#parquet) notes that lists must use the 3-level representation. By default, Hudi was producing 2-level lists in the parquet files at the time that the project was created. We saw that this meant that some engines were unable to read the data at the time.

GitHub link: https://github.com/apache/incubator-xtable/discussions/770#discussioncomment-15254751

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org