You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (Jira)" <ji...@apache.org> on 2020/04/18 08:29:00 UTC
[jira] [Commented] (HIVE-22928) Allow hive.exec.stagingdir to be a
fully qualified directory name
[ https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086374#comment-17086374 ]
Hive QA commented on HIVE-22928:
--------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 56s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 24 new + 83 unchanged - 0 fixed = 107 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 16s{color} | {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21740/dev-support/hive-personality.sh |
| git revision | master / 15ebf9e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21740/yetus/diff-checkstyle-ql.txt |
| asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-21740/yetus/patch-asflicense-problems.txt |
| modules | C: ql U: ql |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21740/yetus.txt |
| Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -----------------------------------------------------------------
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
> Affects Versions: 3.1.2
> Reporter: Thomas Poepping
> Assignee: Thomas Poepping
> Priority: Minor
> Attachments: HIVE-22928.2.patch, HIVE-22928.3.patch, HIVE-22928.4.patch, HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory name that, for operations like {{insert}} or {{insert overwrite}}, will be placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very large (greater than the capacity of the HDFS cluster, as mentioned in a comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their staging directory to be an explicit blobstore path (or any filesystem path), rather than relying on Hive to intelligently build the blobstore path based on an interpretation of the job. We may lose locality guarantees, but because renames are just as expensive on blobstores no matter what the prefix is, this isn't considered a terribly large loss (assuming only blobstore customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all staging directories. For instances where the configured stagingdir scheme is not the same as the scheme for the table location, the default stagingdir configuration is used. This avoids a cross-filesystem rename, which is impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)