You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:29:34 UTC
[I] Efficiently reading hudi tables through spark-shell [hudi]
hudi-bot opened a new issue, #14566:
URL: https://github.com/apache/hudi/issues/14566
[~uditme] Created this ticket to track some discussion on read/query path of spark with Hudi tables.
My understanding is that when you read Hudi tables through spark-shell, some of your queries are slower due to some sequential activity performed by spark when interacting with Hudi tables (even with spark.sql.hive.convertMetastoreParquet which can give you the same data reading speed and all the vectorization benefits). Is this slowness observed during spark query planning ? Can you please elaborate on this ?
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-829
- Type: Task
---
## Comments
22/Apr/20 23:54;uditme;[~nishith29] Thanks for creating the ticket. So the issue I was talking about happens becuase of [HoodieROTablePathFilter|https://github.com/apache/incubator-hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java] which we pass to spark to filter out the latest commit files for copy on write use-case. When we create the *parquet* datasource underneath for reading, Spark uses [InMemoryFileIndex|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala] to maintain an index with list of files to read from the path passed to the data source. Here while the listing of files is parallelized by Spark, the filter however is applied sequentially on the all the file one by one [here|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala#L376]. Here also there are some ca
veats, if the number of paths to be listed < *sparkSession.sessionState.conf.parallelPartitionDiscoveryThreshold* Spark optimizes to do serial listing and the filters are all evaluated sequentially across all partitions at the driver. If however its > *sparkSession.sessionState.conf.parallelPartitionDiscoveryThreshold* then listing across paths is parallelized and filter is applied sequentially at the executors but still sequentially for each file in the path. We have seen this as being a bottleneck with S3 atleast. While ideally this is something that can improved inside of spark by parallelizing the filtering further, however at this point I need to dive further into filter code to understand if there is something that can be done to make it faster from Hudi side.;;;
---
22/Apr/20 23:59;uditme;You may also want to look at my implementation of custom relation in Spark to read bootstrapped tables [https://github.com/apache/incubator-hudi/pull/1475/files#diff-f14ac7b3cff88313d650b01a56a2b8f8R191] . Here I am building my own file index using spark's InMemoryFileIndex, but the filtering part is just one operation now because once I have all the files, the hudi filesystem view is created just once to get latest files. Its still work in progress and I am yet to see how fast this is going to be. We can consider moving to a place where our reading in spark happens through our relations, and underneath we use the native readers.
In practice the above data source implementation I am working on would work for both bootstrapped and non-bootstrapped tables as well. So once this is working and stable would be nice to do some benchmarking for regular (non-bootstrapped) tables as well and compare its read performance against standard parquet data source with filters.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Efficiently reading hudi tables through spark-shell [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14566:
URL: https://github.com/apache/hudi/issues/14566#issuecomment-4795195332
Closing for inactivity. JIRA: [HUDI-829](https://issues.apache.org/jira/browse/HUDI-829). This issue has had no activity for over two years and the GitHub thread has no replies. If this is still relevant, please open a fresh issue with reproduction details on a recent release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Efficiently reading hudi tables through spark-shell [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14566: Efficiently reading hudi tables through spark-shell
URL: https://github.com/apache/hudi/issues/14566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org