You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:36:07 UTC
[I] spark datasource load path format is confused for snapshot and increment read mode [hudi]
hudi-bot opened a new issue, #14675:
URL: https://github.com/apache/hudi/issues/14675
as spark datasource read hudi table
1、snapshot mode
{code:java}
val readHudi = spark.read.format("org.apache.hudi").load(basePath + "/*");
should add "/*" ,otherwise will fail, because in org.apache.hudi.DefaultSource.
createRelation() will use fs.globStatus(). if do not have "/*" will not get .hoodie and default dir
val globPaths = HoodieSparkUtils.checkAndGlobPathIfNecessary(allPaths, fs){code}
2、increment mode
both basePath and basePath + "/*" is ok.This is because in org.apache.hudi.DefaultSource
DataSourceUtils.getTablePath can support both the two format.
{code:java}
val incViewDF = spark.read.format("org.apache.hudi").
option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL).
option(BEGIN_INSTANTTIME_OPT_KEY, beginTime).
option(END_INSTANTTIME_OPT_KEY, endTime).
load(basePath){code}
{code:java}
val incViewDF = spark.read.format("org.apache.hudi").
option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL).
option(BEGIN_INSTANTTIME_OPT_KEY, beginTime).
option(END_INSTANTTIME_OPT_KEY, endTime).
load(basePath + "/*")
{code}
as increment mode and snapshot mode not coincide, user will confuse .Also load use basepath +"/*" *or "/***/*"* is confuse. I know this is to support partition.
but i think this api will more clear for user
{code:java}
partition = "year = '2019'"
spark.read .format("hudi") .load(path) .where(partition) {code}
```
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1307
- Type: Improvement
---
## Comments
30/Sep/20 06:37;309637554;[~vinoth] , [~xleesf] what do you think about this ?:D;;;
---
02/Oct/20 21:51;vinoth;[~309637554] makes sense to make them consistent. go for it!;;;
---
06/Oct/20 09:21;vho;Hi [~309637554] and [~vinoth],
I'm able to query hudi table (SNAPSHOT_QUERY) using just the base path. The trick is to use Hive-style partitioning i.e. part=value for the directory and Spark should be able to figure out the partition automatically.
Using plain value as partitions (e.g. 2020/12/12) has a few drawbacks. First, globing the files can be quite expensive. Second, Spark will not be able to infer the partition schema and therefore query will not be using partition filters.
{code:java}
val cow = spark.read.format("hudi").option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_SNAPSHOT_OPT_VAL).load(cowTablePathLocal + "/*/*/*/*")
cow.createOrReplaceTempView("cow")
spark.sql("select * from cow where year = 2020").explain
== Physical Plan ==
*(1) Project [_hoodie_commit_time#238, _hoodie_commit_seqno#239, _hoodie_record_key#240, _hoodie_partition_path#241, _hoodie_file_name#242, timestamp#243, value#244L, year#245, month#246, day#247, ts#248, dt#249, id#250]
+- *(1) Filter (isnotnull(year#245) && (year#245 = 2020))
+- *(1) FileScan parquet [_hoodie_commit_time#238,_hoodie_commit_seqno#239,_hoodie_record_key#240,_hoodie_partition_path#241,_hoodie_file_name#242,timestamp#243,value#244L,year#245,month#246,day#247,ts#248,dt#249,id#250] Batched: true, Format: Parquet, Location: InMemoryFileIndex[file:/tmp/hoodie/hoodie_streaming_cow/.hoodie/.aux/.bootstrap/.partitions, file..., PartitionFilters: [], PushedFilters: [IsNotNull(year), EqualTo(year,2020)], ReadSchema: struct<_hoodie_commit_time:string,_hoodie_commit_seqno:string,_hoodie_record_key:string,_hoodie_p.. {code}
example
;;;
---
10/Oct/20 16:57;309637554;[~vho] thanks for you feedback. it will be optimized;;;
---
26/Sep/21 18:34;xushiyan;[~309637554] Any update on this improvement? definitely useful to align on the pattern. In fact, today to enable HoodieFileIndex, user should avoid passing in glob path. Is it still necessary to keep glob path pattern around?;;;
---
27/Sep/21 01:10;309637554;[~xushiyan] hello , recently i am focus on ingest kafka data using hudi with clustering, and the online & offline analytics workload schedule on k8s. So this issue have not update.
I think we can keep glob path pattern around, but increment mode and snapshot mode can unified.:D;;;
---
22/Feb/22 15:16;dave_hagman;[~vinoth] [~309637554] [~xushiyan] I'd like to +1 what [~vho] said. We heavily use hive-style partitioning w/ glob paths in order to allow spark to optimize the plan for us.
I wanted to ensure that this issue is not recommending the removal of support for queries such as these:
{code:java}
# Single load
spark.read \
.format("org.apache.hudi") \
.option("hoodie.datasource.query.type", "snapshot") \
.option("hoodie.file.index.enable", False)\
.load("file://example/hudi-table/year=2022/month=2/day=*/hour=*")
# Load multiple years
read_paths = [f"{datalake_uri}/year=2020/month=*/day=*/hour=*", f"{datalake_uri}/year=2021/month=*/day=*/hour=*", ...]
base_df = spark.read \
.format("org.apache.hudi") \
.option("hoodie.datasource.query.type", "snapshot") \
.option("hoodie.file.index.enable", False)\
.option("hoodie.datasource.read.paths", ",".join(read_paths)) \
.load(){code}
In these examples we have a table partitioned by year/month/day/hour.
I am currently testing Hudi version 0.10.x and getting errors when constructing queries the way we currently do with hudi 0.9.0. Have changes already been made to the way the default datasource parses glob patterns?
Example error:
{code:java}
java.lang.NullPointerException
at org.apache.hudi.HoodieSparkUtils$.$anonfun$globPath$2(HoodieSparkUtils.scala:82)
at org.apache.hudi.HoodieSparkUtils$.$anonfun$globPath$2$adapted(HoodieSparkUtils.scala:78)
at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:256)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
at scala.collection.mutable.ArrayOps$ofRef.filterImpl(ArrayOps.scala:198)
at scala.collection.TraversableLike.filterNot(TraversableLike.scala:355)
at scala.collection.TraversableLike.filterNot$(TraversableLike.scala:355)
at scala.collection.mutable.ArrayOps$ofRef.filterNot(ArrayOps.scala:198)
at org.apache.hudi.HoodieSparkUtils$.$anonfun$globPath$1(HoodieSparkUtils.scala:78)
at scala.Option.map(Option.scala:230)
at org.apache.hudi.HoodieSparkUtils$.globPath(HoodieSparkUtils.scala:77)
at org.apache.hudi.HoodieSparkUtils$.globPathIfNecessary(HoodieSparkUtils.scala:97)
at org.apache.hudi.HoodieSparkUtils$.$anonfun$checkAndGlobPathIfNecessary$1(HoodieSparkUtils.scala:111)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.hudi.HoodieSparkUtils$.checkAndGlobPathIfNecessary(HoodieSparkUtils.scala:109)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:97)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:69)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
... {code}
;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] spark datasource load path format is confused for snapshot and increment read mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14675:
URL: https://github.com/apache/hudi/issues/14675#issuecomment-4795224268
Closing for inactivity. JIRA: [HUDI-1307](https://issues.apache.org/jira/browse/HUDI-1307). This issue has had no activity for over two years and the GitHub thread has no replies. If this is still relevant, please open a fresh issue with reproduction details on a recent release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] spark datasource load path format is confused for snapshot and increment read mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14675: spark datasource load path format is confused for snapshot and increment read mode
URL: https://github.com/apache/hudi/issues/14675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org