You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/13 00:29:35 UTC
[GitHub] [hudi] alexeykudinkin commented on issue #6174: Hudi Read Performance : Partition pruning not happening when reading Hudi table
alexeykudinkin commented on issue #6174:
URL: https://github.com/apache/hudi/issues/6174#issuecomment-1244742505
@tarunguptanit would it be possible for you to try out Hudi 0.12?
To explain a little bit what you might be observing:
- First of all in your case you don't rely on partition-pruning, instead you're _directly_ reading one of the partitions by providing tha sub-path w/in the table. While it's somewhat similar to partition pruning behavior this distinctly different mechanism in terms of implementation (inside both Spark and Hudi)
- The reason why you see 1500 tasks being spin'd is b/c even though you're reading one particular partition Hudi currently will be doing file-listing of the whole table (file-listing means that we will just list the files in the table, but we won't be reading the whole table). This is a known issue and there's an effort underway to revisit that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org