You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2025/10/22 09:35:19 UTC
[PR] [SPARK-28098][SQL]Support read hive table when location have multi-level non partitioned paths" [spark]
zhengchenyu opened a new pull request, #52694:
URL: https://github.com/apache/spark/pull/52694
### What changes were proposed in this pull request?
Get data files by prefix matching, then support reading tables with multi-level partition directories.
Although partition matching does introduce additional computation, as far as I know, for TEZ UNION, the location of non-partitioned tables and the partition location of partitioned tables typically don't contain many directories, so this additional computation is negligible.
For situations without multiple levels of directories, the impact is minimal.
### Why are the changes needed?
When table location or partition location have multi-level non partitioned paths. In particular, TEZ will generate the `HIVE_UNION_SUBDIR_1` directory. Spark cannot read the real contents.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit test, and real job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-28098][SQL] Supports reading hive tables when there are subdirectories under the table or partition location. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52694:
URL: https://github.com/apache/spark/pull/52694#issuecomment-3838063854
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-28098][SQL] Supports reading hive tables when there are subdirectories under the table or partition location. [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #52694: [SPARK-28098][SQL] Supports reading hive tables when there are subdirectories under the table or partition location.
URL: https://github.com/apache/spark/pull/52694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-28098][SQL]Support read hive table when location have multi-level non partitioned paths [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu closed pull request #52694: [SPARK-28098][SQL]Support read hive table when location have multi-level non partitioned paths
URL: https://github.com/apache/spark/pull/52694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-28098][SQL]Support read hive table when location have multi-level non partitioned paths [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #52694:
URL: https://github.com/apache/spark/pull/52694#issuecomment-3431386118
I know some pr try to solve this problem.
* #32679 I found that here can not infer partition for non-catalog table.
* #32202 only consider non-partitioned table.
So I submit this new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-28098][SQL] Supports reading hive tables when there are subdirectories under the table or partition location. [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #52694:
URL: https://github.com/apache/spark/pull/52694#issuecomment-3446384514
@cloud-fan @dongjoon-hyun @HyukjinKwon Can you please review this PR? I think it is more reasonable to read files under subdirectories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org