You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rajesh Balamohan <ra...@gmail.com> on 2016/07/26 02:50:50 UTC
Review Request 50434: HIVE-14323: Reduce number of FS permissions and
redundant FS operations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50434/
-----------------------------------------------------------
Review request for hive, Ashutosh Chauhan and Chris Nauroth.
Bugs: HIVE-14323
https://issues.apache.org/jira/browse/HIVE-14323
Repository: hive-git
Description
-------
HIVE-14323: Reduce number of FS permissions and redundant FS operations
1. Honor HIVE_WAREHOUSE_SUBDIR_INHERIT_PERMS when getting stage directory in Context and in mkdir() in FileUtils
2. Reduce FS calls like exists()-->delete() to just delete().
3. Similar to #2 in Hive.replaceFiles()
4. Refactor Hive.trashFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
5. Refactor Hive.removeTempOrDuplicateFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java 23fcc8a
ql/src/java/org/apache/hadoop/hive/ql/Context.java ec5d693
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java cadda8f
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd
Diff: https://reviews.apache.org/r/50434/diff/
Testing
-------
Thanks,
Rajesh Balamohan
Re: Review Request 50434: HIVE-14323: Reduce number of FS permissions
and redundant FS operations
Posted by Rajesh Balamohan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50434/
-----------------------------------------------------------
(Updated Aug. 1, 2016, 8:32 a.m.)
Review request for hive, Ashutosh Chauhan and Chris Nauroth.
Bugs: HIVE-14323
https://issues.apache.org/jira/browse/HIVE-14323
Repository: hive-git
Description
-------
HIVE-14323: Reduce number of FS permissions and redundant FS operations
1. Honor HIVE_WAREHOUSE_SUBDIR_INHERIT_PERMS when getting stage directory in Context and in mkdir() in FileUtils
2. Reduce FS calls like exists()-->delete() to just delete().
3. Similar to #2 in Hive.replaceFiles()
4. Refactor Hive.trashFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
5. Refactor Hive.removeTempOrDuplicateFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java ac08ab3
ql/src/java/org/apache/hadoop/hive/ql/Context.java ec5d693
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java cadda8f
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd
Diff: https://reviews.apache.org/r/50434/diff/
Testing
-------
Thanks,
Rajesh Balamohan
Re: Review Request 50434: HIVE-14323: Reduce number of FS permissions
and redundant FS operations
Posted by Chris Nauroth <cn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50434/#review143565
-----------------------------------------------------------
Ship it!
+1 (non-binding). Thank you, Rajesh.
- Chris Nauroth
On July 26, 2016, 2:50 a.m., Rajesh Balamohan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50434/
> -----------------------------------------------------------
>
> (Updated July 26, 2016, 2:50 a.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Chris Nauroth.
>
>
> Bugs: HIVE-14323
> https://issues.apache.org/jira/browse/HIVE-14323
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-14323: Reduce number of FS permissions and redundant FS operations
>
> 1. Honor HIVE_WAREHOUSE_SUBDIR_INHERIT_PERMS when getting stage directory in Context and in mkdir() in FileUtils
> 2. Reduce FS calls like exists()-->delete() to just delete().
> 3. Similar to #2 in Hive.replaceFiles()
> 4. Refactor Hive.trashFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
> 5. Refactor Hive.removeTempOrDuplicateFiles() so that it can make use of FileStatus information if available (s3 FS lookups are expensive)
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java 23fcc8a
> ql/src/java/org/apache/hadoop/hive/ql/Context.java ec5d693
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java cadda8f
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd
>
> Diff: https://reviews.apache.org/r/50434/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rajesh Balamohan
>
>