You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ofinchuk-bloomberg (via GitHub)" <gi...@apache.org> on 2024/02/15 18:47:17 UTC
[I] [SUPPORT] Can't read a table with timestamp based partition key generator [hudi]
ofinchuk-bloomberg opened a new issue, #10678:
URL: https://github.com/apache/hudi/issues/10678
Can't read a table which was created using TimestampBasedKeyGenerator Or CustomKeyGenerator for timestamp partition.
Issue is that `ts` remains Long type, while _hoodie_partition_path is formed as a String, so Simple operation to read doesn't work and throws Exception
**To Reproduce**
Steps to reproduce the behavior:
`
import org.apache.spark.sql.{SaveMode, SparkSession}
object SprkDemo {
def main(args:Array[String]): Unit ={
val spark = SparkSession.builder()
.master("local[1]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.appName("SparkByExample")
.getOrCreate();
import spark.implicits._
spark.createDataset(List(("id1","name1", System.currentTimeMillis()),
("id2","name2",(System.currentTimeMillis()+1))
))
.toDF("id","name","ts")
.write
.format("hudi")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.CustomKeyGenerator")
.option("hoodie.datasource.write.partitionpath.field", "ts:timestamp")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombined.field", "name")
.option("hoodie.table.name", "hudi_cow2")
.option("hoodie.keygen.timebased.timestamp.type", "EPOCHMILLISECONDS")
.option("hoodie.keygen.timebased.output.dateformat", "yyyyMMdd-HH")
.mode(SaveMode.Overwrite)
.save("/Users/ofinchuk/tools/workspace/hudi/hudi_cow2")
spark.read.parquet("/Users/ofinchuk/tools/workspace/hudi/hudi_cow2/2*")
.show()
spark.read.format("hudi")
.option("hoodie.schema.on.read.enable","true")
.load("/Users/ofinchuk/tools/workspace/hudi/hudi_cow2/")
.show()
}
}
`
when reading parquet I see next data:
`
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+-------------+----------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path| _hoodie_file_name| id| name| ts| date|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+-------------+----------------+
| 20240214184652987|20240214184652987...| id1| 20240214-18|9d4eb7eb-847a-4e1...|id1|name1|1707954411089|2024-02-14 15:00|
| 20240214184652987|20240214184652987...| id2| 20240214-18|9d4eb7eb-847a-4e1...|id2|name2|1707954411090|2024-02-14 15:01|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+-------------+----------------+
`
**Expected behavior**
Table should be read successfully into spark dataframe
**Environment Description**
I use spark 3.3.3 and hudi-spark3.3-bundle_2.12:0.14.1 in local environment
* Running on Docker? (yes/no) :no
**Stacktrace**
```
Exception in thread "main" java.lang.RuntimeException: Failed to cast value '20240214-18' to 'LongType' for partition column 'ts'
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil$.$anonfun$parsePartition$3(Spark3ParsePartitionUtil.scala:78)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil$.$anonfun$parsePartition$2(Spark3ParsePartitionUtil.scala:71)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil$.parsePartition(Spark3ParsePartitionUtil.scala:69)
at org.apache.hudi.HoodieSparkUtils$.parsePartitionPath(HoodieSparkUtils.scala:280)
at org.apache.hudi.HoodieSparkUtils$.parsePartitionColumnValues(HoodieSparkUtils.scala:264)
at org.apache.hudi.SparkHoodieTableFileIndex.doParsePartitionColumnValues(SparkHoodieTableFileIndex.scala:401)
at org.apache.hudi.BaseHoodieTableFileIndex.parsePartitionColumnValues(BaseHoodieTableFileIndex.java:364)
at org.apache.hudi.BaseHoodieTableFileIndex.lambda$listPartitionPaths$7(BaseHoodieTableFileIndex.java:333)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.apache.hudi.BaseHoodieTableFileIndex.listPartitionPaths(BaseHoodieTableFileIndex.java:336)
at org.apache.hudi.BaseHoodieTableFileIndex.getAllQueryPartitionPaths(BaseHoodieTableFileIndex.java:216)
at org.apache.hudi.SparkHoodieTableFileIndex.listMatchingPartitionPaths(SparkHoodieTableFileIndex.scala:219)
at org.apache.hudi.HoodieFileIndex.getFileSlicesForPrunedPartitions(HoodieFileIndex.scala:282)
at org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:211)
at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:151)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Can't read a table with timestamp based partition key generator [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #10678:
URL: https://github.com/apache/hudi/issues/10678#issuecomment-4843425406
Hi @ofinchuk-bloomberg
I have provided an alternative solution for your issue. If this looks okay, we can close the issue.
```python
table_path = "/tmp/hudi_10678_simple_redesign"
df = df.withColumn("part_hour", F.date_format(F.timestamp_millis(F.col("ts")), "yyyyMMdd-HH"))
df.printSchema()
print("Workaround — dedicated string partition column")
(df.write.format("hudi")
.option("hoodie.table.name", "t10678_d")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.SimpleKeyGenerator")
.option("hoodie.datasource.write.partitionpath.field", "part_hour") # native STRING
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombine.field", "name")
.option("hoodie.datasource.write.hive_style_partitioning", "true")
.mode("overwrite")
.save(table_path))
spark.read.format("hudi").load(table_path).show(truncate=False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Can't read a table with timestamp based partition key generator [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10678:
URL: https://github.com/apache/hudi/issues/10678#issuecomment-1970661775
@ofinchuk-bloomberg This is same issue as https://issues.apache.org/jira/browse/HUDI-4818 .
It is still reproducible on Will try to get this fix prioritized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Can't read a table with timestamp based partition key generator [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #10678:
URL: https://github.com/apache/hudi/issues/10678#issuecomment-1974143434
@mansipp is this similar to what we saw when reading Hudi tables via Trino?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Can't read a table with timestamp based partition key generator [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10678:
URL: https://github.com/apache/hudi/issues/10678#issuecomment-1948179771
@ofinchuk-bloomberg Thanks for raising this. I was able to reproduce. Checking it is same as one other known issue which was fixed by https://github.com/apache/hudi/pull/9863 which was later reverted due to little performance regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org