You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:31:48 UTC
[I] Support YYYY/MM/DD partition format with hive [hudi]
hudi-bot opened a new issue, #17282:
URL: https://github.com/apache/hudi/issues/17282
Currently a format like YYYY/MM/DD fails when syncing with hive. The Jira aims to add a fix so that such a format is supported.
Steps to reproduce: The table created below uses a custom keygen with combination of simple and timestamp keygen. Timestamp keygen produces an output of format - YYYY/MM/DD
{code:java}
import org.apache.hudi.HoodieSparkUtils
import org.apache.hudi.common.config.TypedProperties
import org.apache.hudi.common.util.StringUtils
import org.apache.hudi.exception.HoodieException
import org.apache.hudi.functional.TestSparkSqlWithCustomKeyGenerator._
import org.apache.hudi.testutils.HoodieClientTestUtils.createMetaClient
import org.apache.hudi.util.SparkKeyGenUtilsimport org.apache.spark.sql.SaveMode
import org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat
import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertTrue}
import org.slf4j.LoggerFactory
val df = spark.sql(
s"""SELECT 1 as id, 'a1' as name, 1.6 as price, 1704121827 as ts, 'cat1' as segment
| UNION
| SELECT 2 as id, 'a2' as name, 10.8 as price, 1704121827 as ts, 'cat1' as segment
| UNION
| SELECT 3 as id, 'a3' as name, 30.0 as price, 1706800227 as ts, 'cat1' as segment
| UNION
| SELECT 4 as id, 'a4' as name, 103.4 as price, 1701443427 as ts, 'cat2' as segment
| UNION
| SELECT 5 as id, 'a5' as name, 1999.0 as price, 1704121827 as ts, 'cat2' as segment
| UNION
| SELECT 6 as id, 'a6' as name, 80.0 as price, 1704121827 as ts, 'cat3' as segment
|""".stripMargin) df.write.format("hudi").option("hoodie.datasource.write.table.type", "MERGE_ON_READ").option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.CustomAvroKeyGenerator").option("hoodie.datasource.write.partitionpath.field", "segment:simple,ts:timestamp").option("hoodie.datasource.write.recordkey.field", "id").option("hoodie.datasource.write.precombine.field", "name").option("hoodie.table.name", "hudi_table_2").option("hoodie.insert.shuffle.parallelism", "1").option("hoodie.upsert.shuffle.parallelism", "1").option("hoodie.bulkinsert.shuffle.parallelism", "1").option("hoodie.keygen.timebased.timestamp.type", "SCALAR").option("hoodie.keygen.timebased.output.dateformat", "yyyy/MM/DD").option("hoodie.keygen.timebased.timestamp.scalar.time.unit", "seconds").mode(SaveMode.Overwrite).save("/user/hive/warehouse/hudi_table_2")
// Sync with hive
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
--jdbc-url jdbc:hive2://hiveserver:10000 \
--user hive \
--pass hive \
--partitioned-by segment,ts \
--base-path /user/hive/warehouse/hudi_table_2 \
--database default \
--table hudi_table_2 \
--partition-value-extractor org.apache.hudi.hive.MultiPartKeysValueExtractor {code}
Hive creation fails now.
{code:java}
2024-10-06 14:33:44,200 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_table_2
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:180)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:547)
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: failed to sync the table hudi_table_2_ro
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:272)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:203)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:177)
... 1 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_table_2_ro
at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:474)
at org.apache.hudi.hive.HiveSyncTool.validateAndSyncPartitions(HiveSyncTool.java:321)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:261)
... 3 more
Caused by: java.lang.IllegalArgumentException: Partition key parts [segment, ts] does not match with partition values [cat1, 2024, 01, 01]. Check partition strategy.
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.getPartitionClause(QueryBasedDDLExecutor.java:191)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.constructAddPartitions(QueryBasedDDLExecutor.java:164)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.addPartitionsToTable(QueryBasedDDLExecutor.java:124)
at org.apache.hudi.hive.HoodieHiveSyncClient.addPartitionsToTable(HoodieHiveSyncClient.java:118)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:516)
at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:470)
... 5 more {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8311
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support YYYY/MM/DD partition format with hive [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17282:
URL: https://github.com/apache/hudi/issues/17282#issuecomment-5505512201
Re-checked against `master` at `a2788eef922e`. This is decidable from source, so amending my earlier note that it was not.
Correcting one thing in that note first: I suggested this was "almost certainly the same fix" as #17283 (HUDI-8312). It is not, and I have posted a retraction there. The two fail for unrelated reasons at different places, and merging them would hide one behind the other.
### Why the sync fails
`/` is Hudi's partition-path separator, so a `yyyy/MM/DD` output format does not produce one partition value containing slashes, it produces **three extra path segments**. The written path for the reproduction is `cat1/2024/01/01`: four segments, against the two fields declared by `--partitioned-by segment,ts`.
The DDL builder resolves the path through the configured extractor and then asserts the counts agree:
https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java#L221-L237
`MultiPartKeysValueExtractor` splits flat on `/`, so it returns `[cat1, 2024, 01, 01]`, four values for two keys, and `checkArgument` throws with exactly the message in the report. Nothing is sent to Hive: this fails inside Hudi while the statement is still being built. (The stack trace in the description shows `:191`; the check is at `:223` on current master.)
Contrast #17283, where `yyyy-MM-DD` yields `cat1/2024-10-01`, two segments for two keys. The arity check passes there, the `ALTER TABLE` is emitted, and Hive is what rejects it, for an unrelated reason (the partition column is typed from the table column, which is numeric).
### The actual gap
The extractors available today each handle one shape, and this configuration falls between them:
- `MultiPartKeysValueExtractor` - N flat segments for N fields. Cannot absorb a multi-segment value.
- `SlashEncodedDayPartitionValueExtractor` - collapses `yyyy/mm/dd` into a single value, but requires the path to be **exactly** three segments and throws otherwise, so it cannot cope with the leading `cat1`.
- `SlashEncodedHourPartitionValueExtractor`, `SinglePartPartitionValueExtractor`, `HiveStylePartitionValueExtractor`, `NonPartitionedExtractor` - none applies.
So the missing case is the **mixed** layout: a `CustomKeyGenerator` with `segment:simple,ts:timestamp` and a slash-bearing date format produces one single-segment field followed by one three-segment field, and no extractor knows that a given field spans more than one segment.
### Fix direction
The information needed is already in the write config: `hoodie.datasource.write.partitionpath.field` says which fields are `timestamp` type, and the output format says how many segments each contributes. An extractor driven by that per-field segment count would handle this and any other multi-segment format without a new class per shape. The cheaper interim option is to reject a slash-bearing `hoodie.keygen.timebased.output.dateformat` at write time when the sync cannot represent it, so the failure lands at configuration time rather than at sync time.
Read from source; I have not reproduced this against a live metastore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support YYYY/MM/DD partition format with hive [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17282:
URL: https://github.com/apache/hudi/issues/17282#issuecomment-5365812951
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8311).
**Findings: needs the reproduction run with a Hive metastore; not decidable from source.**
The report is specific - a custom keygen combining `SimpleKeyGenerator` with `TimestampBasedKeyGenerator` produces partition values formatted `YYYY/MM/DD`, and syncing those to Hive fails. The description carries a full Scala reproduction built on the `TestSparkSqlWithCustomKeyGenerator` helpers, which is enough to run directly.
Whether such a partition value round-trips through Hive depends on partition-path escaping and on how the synced Hive partition spec is constructed from the path - not something the source settles either way.
**This is almost certainly the same fix as #17283 (HUDI-8312)**, which reports the identical failure for the other separator. Suggest merging them, or at least fixing them together, since a solution that handles one separator and not the other would be arbitrary. Worth also checking the other common formats (`YYYYMMDD`, `YYYY-MM-DD-HH`) in the same pass so this does not recur one separator at a time.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org