You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:53:15 UTC
[I] Fix timestamp based partitioning and drop partition support with 0.15.0 [hudi]
hudi-bot opened a new issue, #17365:
URL: https://github.com/apache/hudi/issues/17365
Based on our analysis, drop partition support is broken in 0.15.0 for multi partition fields.
For nested field, it is swapping a field with the same name but different path with the partition value
For timestamp issue, the field gets replaced with the partition value instead of the value in the file (for example: {{{}timestamp_micros_nullable_field":"2025-01-25T00:00:00.000Z"{}}})
Also seeing a regression on drop partition where the dropped partition is still being read
The replace commit is not being written correctly in 0.15.0, the {{partitionToReplaceFileIds}} contains a map with an empty list instead of the filegroup ids for the partition
We need a fix for 0.15.0.
1.0 works fine. See the test script https://gist.github.com/codope/dfb87d35112ddbb7f207ad3f52320071
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8928
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 0.15.1
- Attachment(s):
- 28/Jan/25 10:44;codope;Screenshot 2025-01-28 at 4.12.49 PM.png;https://issues.apache.org/jira/secure/attachment/13074315/Screenshot+2025-01-28+at+4.12.49%E2%80%AFPM.png
---
## Comments
28/Jan/25 10:45;codope;With 1.0, ran a simple DELETE_PARTITION test and I can see that `partitionToReplaceFileIds` is populated correctly.
!Screenshot 2025-01-28 at 4.12.49 PM.png|width=990,height=106!
SQL also works: `spark.sql(s"alter table $tableName drop partition (dt='2021-10-01')")`;;;
---
28/Jan/25 12:18;codope;Ran [TestSparkSqlWithCustomKeyGenerator|https://github.com/apache/hudi/blob/02472c91aac1892d76602795c3f816b58e9c90f7/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala#L255] - could not repro.
{code:java}
val df = spark.sql(
s"""SELECT 1 as id, 'a1' as name, 1.6 as price, 1704121827 as ts, 'cat1' as segment
| UNION
| SELECT 2 as id, 'a2' as name, 10.8 as price, 1704121827 as ts, 'cat1' as segment
| UNION
| SELECT 3 as id, 'a3' as name, 30.0 as price, 1706800227 as ts, 'cat1' as segment
| UNION
| SELECT 4 as id, 'a4' as name, 103.4 as price, 1701443427 as ts, 'cat2' as segment
| UNION
| SELECT 5 as id, 'a5' as name, 1999.0 as price, 1704121827 as ts, 'cat2' as segment
| UNION
| SELECT 6 as id, 'a6' as name, 80.0 as price, 1704121827 as ts, 'cat3' as segment
|""".stripMargin)
df.write.format("hudi")
.option("hoodie.datasource.write.table.type", tableType)
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.CustomKeyGenerator")
.option("hoodie.datasource.write.partitionpath.field", "ts:timestamp,segment:simple")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombine.field", "name")
.option("hoodie.table.name", tableName)
.option("hoodie.insert.shuffle.parallelism", "1")
.option("hoodie.upsert.shuffle.parallelism", "1")
.option("hoodie.bulkinsert.shuffle.parallelism", "1")
.option("hoodie.keygen.timebased.timestamp.type", "SCALAR")
.option("hoodie.keygen.timebased.output.dateformat", "yyyyMM")
.option("hoodie.keygen.timebased.timestamp.scalar.time.unit", "seconds")
.mode(SaveMode.Overwrite)
.save(tablePath)
spark.read.format("hudi").load(tablePath).show(false)
+-------------------+---------------------+------------------+----------------------+--------------------------------------------------------------------------+---+----+------+----------+-------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |name|price |ts |segment|
+-------------------+---------------------+------------------+----------------------+--------------------------------------------------------------------------+---+----+------+----------+-------+
|20250128120421899 |20250128120421899_0_0|2 |202401/cat1 |390ee34b-6466-46fb-99da-9c7010d87413-0_0-181-278_20250128120421899.parquet|2 |a2 |10.8 |1704121827|cat1 |
|20250128120421899 |20250128120421899_0_1|1 |202401/cat1 |390ee34b-6466-46fb-99da-9c7010d87413-0_0-181-278_20250128120421899.parquet|1 |a1 |1.6 |1704121827|cat1 |
|20250128120421899 |20250128120421899_2_0|6 |202401/cat3 |0b744252-3504-47f6-83b5-36a8ad1d9bbd-0_2-181-280_20250128120421899.parquet|6 |a6 |80.0 |1704121827|cat3 |
|20250128120421899 |20250128120421899_4_0|4 |202312/cat2 |99e01331-1443-4058-bca7-c35cd56b3c77-0_4-181-282_20250128120421899.parquet|4 |a4 |103.4 |1701443427|cat2 |
|20250128120421899 |20250128120421899_3_0|3 |202402/cat1 |3b33ac51-c756-4b04-af35-cb358f9ba80a-0_3-181-281_20250128120421899.parquet|3 |a3 |30.0 |1706800227|cat1 |
|20250128120421899 |20250128120421899_1_0|5 |202401/cat2 |3c8ec017-883d-4ba5-b550-437ee3cc2246-0_1-181-279_20250128120421899.parquet|5 |a5 |1999.0|1704121827|cat2 |
+-------------------+---------------------+------------------+----------------------+--------------------------------------------------------------------------+---+----+------+----------+-------+{code}
As you can see, `hoodie_partition_path` is in the specified output format, while `ts` field is same as the input data.;;;
---
28/Jan/25 12:36;codope;Timestamp partitioning issue reproducible with 0.15.0. Here's a simple script to repro the issue - [https://gist.github.com/codope/dfb87d35112ddbb7f207ad3f52320071];;;
---
28/Jan/25 15:22;codope;The replacecommit looks fine in both 0.15.0 and master branch. All tests updated in this gist - https://gist.github.com/codope/dfb87d35112ddbb7f207ad3f52320071;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix timestamp based partitioning and drop partition support with 0.15.0 [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17365:
URL: https://github.com/apache/hudi/issues/17365#issuecomment-4892474696
**Sample Reproducible Code:**
```python
table_name = "ts_multi_table"
table_path = f"s3://warehouse/{table_name}"
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType, LongType
opts = {
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.CustomKeyGenerator",
"hoodie.datasource.write.partitionpath.field": "ts:timestamp,segment:simple", # multi-field, timestamp + simple
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "name",
"hoodie.table.name": table_name,
"hoodie.keygen.timebased.timestamp.type": "SCALAR",
"hoodie.keygen.timebased.output.dateformat": "yyyyMM",
"hoodie.keygen.timebased.timestamp.scalar.time.unit": "seconds",
"hoodie.metadata.enable": "false",
}
schema = StructType([
StructField("id", IntegerType(), False), StructField("name", StringType(), False),
StructField("price", DoubleType(), False), StructField("ts", LongType(), False),
StructField("segment", StringType(), False)])
rows = [(1, "a1", 1.6, 1704121827, "cat1"), (2, "a2", 10.8, 1704121827, "cat1"),
(3, "a3", 30.0, 1706800227, "cat1"), (4, "a4", 103.4, 1701443427, "cat2"),
(5, "a5", 1999.0, 1704121827, "cat2"), (6, "a6", 80.0, 1704121827, "cat3")]
df = spark.createDataFrame(rows, schema)
INPUT_TS = sorted({1704121827, 1706800227, 1701443427})
try:
df.write.format("hudi").options(**opts).option("hoodie.datasource.write.operation", "insert") \
.mode("overwrite").save(table_path)
r1 = spark.read.format("hudi").load(table_path)
ts_vals = sorted({row["ts"] for row in r1.select("ts").collect()})
parts = sorted({row["_hoodie_partition_path"] for row in r1.select("_hoodie_partition_path").collect()})
print(">>> ts values read = %s expected original epochs = %s" % (ts_vals, INPUT_TS))
print(">>> partitions = %s" % parts)
ts_ok = (ts_vals == INPUT_TS) # (A) data value preserved, not replaced by partition value
# (B) drop one partition and confirm it disappears
drop_part = parts[0]
before = r1.count()
df.write.format("hudi").options(**opts) \
.option("hoodie.datasource.write.operation", "delete_partition") \
.option("hoodie.datasource.write.partitions.to.delete", drop_part) \
.mode("append").save(table_path)
r2 = spark.read.format("hudi").load(table_path)
after = r2.count()
remaining = sorted({row["_hoodie_partition_path"] for row in r2.select("_hoodie_partition_path").collect()})
print(">>> dropped=%s before=%d after=%d remaining=%s" % (drop_part, before, after, remaining))
drop_ok = (drop_part not in remaining) and (after < before)
print(">>> ts_value_correct=%s drop_partition_ok=%s %s"
% (ts_ok, drop_ok, "PASS" if (ts_ok and drop_ok) else "FAIL"))
except Exception as e:
print(">>> RESULT=FAILED :: %s :: %s" % (type(e).__name__, str(e)[:300]))
```
**Hudi 0.15.0 Output:**
```sh
>>> ts values read = [202312, 202401, 202402] expected original epochs = [1701443427, 1704121827, 1706800227]
>>> partitions = ['202312/cat2', '202401/cat1', '202401/cat2', '202401/cat3', '202402/cat1']
26/07/06 11:45:47 WARN HoodieSparkSqlWriterInternal: Closing write client
>>> dropped=202312/cat2 before=6 after=5 remaining=['202401/cat1', '202401/cat2', '202401/cat3', '202402/cat1']
>>> ts_value_correct=False drop_partition_ok=True FAIL
```
**Hudi 1.2.0 Output:**
```sh
>>> ts values read = [1701443427, 1704121827, 1706800227] expected original epochs = [1701443427, 1704121827, 1706800227]
>>> partitions = ['202312/cat2', '202401/cat1', '202401/cat2', '202401/cat3', '202402/cat1']
>>> dropped=202312/cat2 before=6 after=5 remaining=['202401/cat1', '202401/cat2', '202401/cat3', '202402/cat1']
>>> ts_value_correct=True drop_partition_ok=True PASS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix timestamp based partitioning and drop partition support with 0.15.0 [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #17365: Fix timestamp based partitioning and drop partition support with 0.15.0
URL: https://github.com/apache/hudi/issues/17365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix timestamp based partitioning and drop partition support with 0.15.0 [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17365:
URL: https://github.com/apache/hudi/issues/17365#issuecomment-4892569201
This issue was reviewed as part of the JIRA-migrated backlog triage and reproduced/validated on the spark-hudi container (Spark 3.4).
Findings: the timestamp-based partitioning defect reproduces on 0.15.0 and is fixed on current Hudi. Test: CustomKeyGenerator with partitionpath.field=ts:timestamp,segment:simple (SCALAR seconds, yyyyMM).
- On 0.15.0, reading back returns the partition-derived value in the ts column (ts=[202312,202401,202402]) instead of the original epochs - the reported 'field replaced with partition value' bug.
- On 1.2.0 the ts column returns the original epochs [1701443427,1704121827,1706800227] - fixed. (delete_partition worked correctly on both in a single-partition-drop check.)
The fix was targeted at 0.15.1, which was never released (0.15.x is EOL - releases go 0.15.0 -> 1.0.0 -> 1.2.0), and 1.0+ already works, so no supported version still exhibits this and there is no backport target.
Closing as fixed on supported versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org