You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Davis Zhang (Jira)" <ji...@apache.org> on 2025/01/11 01:03:00 UTC
[jira] [Comment Edited] (HUDI-8817) Hudi 1.0.0 cannot create older version table
[ https://issues.apache.org/jira/browse/HUDI-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17912036#comment-17912036 ]
Davis Zhang edited comment on HUDI-8817 at 1/11/25 1:02 AM:
------------------------------------------------------------
I'm able to reproduce this issue with
{code:java}
${SPARK_HOME}/bin/spark-sql --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar' \
--conf 'spark.sql.catalogImplementation=in-memory'
spark-sql (default)> CREATE TABLE hudi_14_table_sql_005 (
> event_id INT,
> event_date STRING,
> event_name STRING,
> event_ts STRING,
> event_type STRING
> ) USING hudi
> OPTIONS(
> type = 'cow', – or 'mor'
> primaryKey = 'event_id,event_date',
> preCombileField = 'event_ts',
> hoodie.write.table.version = 6
> )
> PARTITIONED BY (event_type)
> LOCATION 'file:///tmp/lakes/observed-default/dd/hudi_14_table_sql_005'; {code}
then checking the hoodie.properties, I got
{code:java}
➜ .hoodie cat hoodie.properties
#Properties saved on 2025-01-10T16:47:23.630882Z
#Fri Jan 10 08:47:23 PST 2025
hoodie.compaction.payload.class=org.apache.hudi.common.model.DefaultHoodieRecordPayload
hoodie.table.initial.version=8
hoodie.table.keygenerator.type=COMPLEX
hoodie.table.type=COPY_ON_WRITE
hoodie.table.partition.fields=event_type
hoodie.archivelog.folder=archived
hoodie.table.create.schema={"type"\:"record","name"\:"hudi_14_table_sql_005_record","namespace"\:"hoodie.hudi_14_table_sql_005","fields"\:[
{"name"\:"event_id","type"\:["int","null"]}
,{"name"\:"event_date","type"\:["string","null"]},{"name"\:"event_name","type"\:["string","null"]},{"name"\:"event_ts","type"\:["string","null"]},{"name"\:"event_type","type"\:["string","null"]}]}
hoodie.timeline.layout.version=2
hoodie.timeline.history.path=history
hoodie.table.checksum=2973677582
hoodie.datasource.write.drop.partition.columns=false
hoodie.record.merge.strategy.id=eeb8d96f-b1e4-49fd-bbf8-28ac514178e5
hoodie.table.name=hudi_14_table_sql_005
hoodie.table.recordkey.fields=event_id,event_date
hoodie.timeline.path=timeline
hoodie.datasource.write.hive_style_partitioning=true
hoodie.database.name=default
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.record.merge.mode=EVENT_TIME_ORDERING
hoodie.table.version=8 {code}
was (Author: JIRAUSER305408):
I'm able to reproduce this issue with
```
${*}{{*}SPARK_HOME{*}}{*}/bin/spark-sql --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar' \
--conf 'spark.sql.catalogImplementation=in-memory'
spark-sql (default)> CREATE TABLE hudi_14_table_sql_005 (
> event_id INT,
> event_date STRING,
> event_name STRING,
> event_ts STRING,
> event_type STRING
> ) USING hudi
> OPTIONS(
> type = 'cow', -- or 'mor'
> primaryKey = 'event_id,event_date',
> preCombileField = 'event_ts',
> hoodie.write.table.version = 6
> )
> PARTITIONED BY (event_type)
> LOCATION 'file:///tmp/lakes/observed-default/dd/hudi_14_table_sql_005';
```
then checking the hoodie.properties, I got
```
*➜* *.hoodie* cat hoodie.properties
#Properties saved on 2025-01-10T16:47:23.630882Z
#Fri Jan 10 08:47:23 PST 2025
hoodie.compaction.payload.class=org.apache.hudi.common.model.DefaultHoodieRecordPayload
hoodie.table.initial.version=8
hoodie.table.keygenerator.type=COMPLEX
hoodie.table.type=COPY_ON_WRITE
hoodie.table.partition.fields=event_type
hoodie.archivelog.folder=archived
hoodie.table.create.schema=\{"type"\:"record","name"\:"hudi_14_table_sql_005_record","namespace"\:"hoodie.hudi_14_table_sql_005","fields"\:[{"name"\:"event_id","type"\:["int","null"]},\{"name"\:"event_date","type"\:["string","null"]},\{"name"\:"event_name","type"\:["string","null"]},\{"name"\:"event_ts","type"\:["string","null"]},\{"name"\:"event_type","type"\:["string","null"]}]}
hoodie.timeline.layout.version=2
hoodie.timeline.history.path=history
hoodie.table.checksum=2973677582
hoodie.datasource.write.drop.partition.columns=false
hoodie.record.merge.strategy.id=eeb8d96f-b1e4-49fd-bbf8-28ac514178e5
hoodie.table.name=hudi_14_table_sql_005
hoodie.table.recordkey.fields=event_id,event_date
hoodie.timeline.path=timeline
hoodie.datasource.write.hive_style_partitioning=true
hoodie.database.name=default
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.record.merge.mode=EVENT_TIME_ORDERING
hoodie.table.version=8
```
> Hudi 1.0.0 cannot create older version table
> --------------------------------------------
>
> Key: HUDI-8817
> URL: https://issues.apache.org/jira/browse/HUDI-8817
> Project: Apache Hudi
> Issue Type: Sub-task
> Affects Versions: 1.0.0
> Reporter: Shawn Chang
> Assignee: Davis Zhang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.1
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When using Hudi 1.0 + backward writer, it still creates version 8 table which cannot be read by an older version Hudi reader.
> Reproduction steps:
> 1. Create the table with SQL, specifying the table version to be 6 (I've tested the same with DF and Hudi 1.0 also cannot create version 6 table)
> {code:java}
> CREATE TABLE hudi_14_table_sql_005 (
> event_id INT,
> event_date STRING,
> event_name STRING,
> event_ts STRING,
> event_type STRING
> ) USING hudi
> OPTIONS(
> type = 'cow', -- or 'mor'
> primaryKey = 'event_id,event_date',
> preCombileField = 'event_ts',
> hoodie.write.table.version = 6
> )
> PARTITIONED BY (event_type)
> LOCATION 's3://<some_bucket>/hudi_14_table_sql_005';
> {code}
> 2. Check `hoodie.properties` under the table's S3 location, and you should see `hoodie.table.version=8`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)