You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/07/29 07:09:33 UTC
[GitHub] [hive] rbalamohan opened a new pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
rbalamohan opened a new pull request #2545:
URL: https://github.com/apache/hive/pull/2545
### What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/HIVE-24546
Fix FS usage
### Why are the changes needed?
Optimised FS usage for objectstores; especially during dynamic partition loads.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
small internal cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2545:
URL: https://github.com/apache/hive/pull/2545#issuecomment-974730521
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2545:
URL: https://github.com/apache/hive/pull/2545#discussion_r756090246
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
##########
@@ -910,12 +910,12 @@ private void updateDPCounters(final FSPaths fsp, final int filesIdx) {
}
private void createDpDirCheckSrc(final Path dpStagingPath, final Path dpFinalPath) throws IOException {
- if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
Review comment:
if we remove the extra argument we get the same method as `createDpDir` - so there must have been a reason for that if in [HIVE-17778](https://github.com/apache/hive/commit/a9f25c0e7ad3f81a9f00f601947a161516e33f1b#diff-75a40b9ea37fdd5cad43bb2690f796ab678e2c7cf92be78f8de150b3f3c0d98dR819) an extra if and these 2 methods were added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #2545:
URL: https://github.com/apache/hive/pull/2545#discussion_r757960160
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
##########
@@ -910,12 +910,12 @@ private void updateDPCounters(final FSPaths fsp, final int filesIdx) {
}
private void createDpDirCheckSrc(final Path dpStagingPath, final Path dpFinalPath) throws IOException {
- if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
Review comment:
Fix should be in "createDpDir" (ref: stacktrace in the ticket). I have corrected this in the latest patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2545:
URL: https://github.com/apache/hive/pull/2545#issuecomment-974730521
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #2545:
URL: https://github.com/apache/hive/pull/2545#discussion_r756085431
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
##########
@@ -910,12 +910,12 @@ private void updateDPCounters(final FSPaths fsp, final int filesIdx) {
}
private void createDpDirCheckSrc(final Path dpStagingPath, final Path dpFinalPath) throws IOException {
- if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
Review comment:
I don't fully understand why we had `dpFinalPath` as a parameter to this method; and why was it also checked before creating the `dpStagingPath` => if `dpFinalPath` was already existed the staging dir was not created....I don't know if that was a bug or intentional....
but now this `dpFinalPath` also become unused; so it could be removed from the function signature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2545:
URL: https://github.com/apache/hive/pull/2545#issuecomment-1025012668
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #2545: HIVE-24546: Avoid unwanted cloud storage call during dynamic partitio…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #2545:
URL: https://github.com/apache/hive/pull/2545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org