You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:17:04 UTC
[I] Using CustomKeyGenerator fails w/ SparkHoodieTableFileIndex [hudi]
hudi-bot opened a new issue, #17225:
URL: https://github.com/apache/hudi/issues/17225
Currently using `CustomKeyGenerator` with the partition-path config \{hoodie.datasource.write.partitionpath.field=ts:timestamp} fails w/
{code:java}
Caused by: java.lang.RuntimeException: Failed to cast value `2022-05-11` to `LongType` for partition column `ts_ms`
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.$anonfun$parsePartition$2(Spark3ParsePartitionUtil.scala:72)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.$anonfun$parsePartition$1(Spark3ParsePartitionUtil.scala:65)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.parsePartition(Spark3ParsePartitionUtil.scala:63)
at org.apache.hudi.SparkHoodieTableFileIndex.parsePartitionPath(SparkHoodieTableFileIndex.scala:274)
at org.apache.hudi.SparkHoodieTableFileIndex.parsePartitionColumnValues(SparkHoodieTableFileIndex.scala:258)
at org.apache.hudi.BaseHoodieTableFileIndex.lambda$getAllQueryPartitionPaths$3(BaseHoodieTableFileIndex.java:190)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.hudi.BaseHoodieTableFileIndex.getAllQueryPartitionPaths(BaseHoodieTableFileIndex.java:193) {code}
This occurs b/c SparkHoodieTableFileIndex produces incorrect partition schema at XXX
where it properly handles only `TimestampBasedKeyGenerator`s but not the other key-generators that might be changing the data-type of the partition-value as compared to the source partition-column (in this case it has `ts` as a long in the source table schema, but it produces partition-value as string)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4818
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0
---
## Comments
28/Feb/24 12:30;adityagoenka;This is still open -
One more GitHub issue reported -
[https://github.com/apache/hudi/issues/10678]
Reproducible code -
```
data = [("id1", "name1", time.time_ns()),
("id2", "name2", time.time_ns()+ 1)]
schema = StructType([
StructField("id", StringType(), True),
StructField("name", StringType(), True),
StructField("ts", LongType(), True)
])
df = spark.createDataFrame(data, schema=schema)
(df.write.format("hudi")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.CustomKeyGenerator")
.option("hoodie.datasource.write.partitionpath.field", "ts:timestamp")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombined.field", "name")
.option("hoodie.table.name", "hudi_cow2")
.option("hoodie.keygen.timebased.timestamp.type", "EPOCHMILLISECONDS")
.option("hoodie.keygen.timebased.output.dateformat", "yyyyMMdd-HH")
.mode("overwrite")
.save(PATH))
df_read_parquet = spark.read.parquet(PATH + "/5*")
df_read_parquet.show()
df_read_hudi = (spark.read.format("hudi")
.option("hoodie.schema.on.read.enable", "true")
.load(PATH))
df_read_hudi.show()
```;;;
---
17/Sep/24 02:31;guoyihua;I have a WIP fix (see the branch below) on making partition pruning work for the timestamp-based partition field with the timestamp or custom key generator. If this issue is related to the partition pruning, I'll push through this fix.
[https://github.com/yihua/hudi/tree/HUDI-fix-timestamp-partition]
;;;
---
15/Oct/24 23:00;linliu;[~yihua], Do you need us to investigate this ticket, and let you know the root cause? Since you already have the PR, should we land it now?;;;
---
16/Oct/24 00:52;linliu;I reproduced the issue on my local, and I confirmed this is related to the partition pruning. Would you please push your fix? [~yihua]
;;;
---
21/Oct/24 16:10;linliu;[~yihua], I applied your fix, but it did not fix the issue. So I have to debug then.;;;
---
21/Oct/24 18:13;linliu;The root cause of the failure is that `.option("hoodie.keygen.timebased.output.dateformat", "yyyyMMdd-HH")`. the out put format of the timestamp contains '\-', which cannot be cast into long type. After removing the '-', it worked.;;;
---
22/Oct/24 01:38;yihua;I think we still need to fix this for timestamp or custom key generator to work properly with the partition pruning.;;;
---
22/Oct/24 05:45;linliu;The description of this ticket does not mention about partition pruning.
Please add property description or create a new ticket.;;;
---
26/Nov/24 19:30;yihua;Let's validate the behavior of the following scenarios on both Hudi 0.14 and Hudi 1.0 to make sure there is no regression:
Key generators:
(1) Timestamp keygen, timestamp partition column (long type), SCALAR timestamp type, yyyy-MM-dd as output format
(2) Custom keygen, simple (string type) and timestamp partition column (long type), SCALAR timestamp type, yyyy-MM-dd as output format
Queries to test:
(1) full table scan
(2) filter for partition with string, e.g., `partition='yyyy-MM-dd'`
(3) filter for partition with timestamp, e.g., `partition between 123 and 456`, where "123" and "456" are the raw values from the partition column.;;;
---
27/Nov/24 20:07;jonvex;Here is the prototype for a fix [https://github.com/jonvex/hudi/tree/fix_timestamp_keygen_vectorized]
in [https://github.com/apache/spark/blob/32232e9ed33bb16b93ad58cfde8b82e0f07c0970/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L206] it removes the partition only filters from the list of filters that that should be run after reading the files. We need to not remove filters for timestamp keygen cols;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Using CustomKeyGenerator fails w/ SparkHoodieTableFileIndex [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed issue #17225: Using CustomKeyGenerator fails w/ SparkHoodieTableFileIndex
URL: https://github.com/apache/hudi/issues/17225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Using CustomKeyGenerator fails w/ SparkHoodieTableFileIndex [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #17225:
URL: https://github.com/apache/hudi/issues/17225#issuecomment-3630293468
### Linked PR(s)
- https://github.com/apache/hudi/pull/6641
- https://github.com/apache/hudi/pull/12165
- https://github.com/apache/hudi/pull/12218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org