You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/12/07 08:13:19 UTC
[GitHub] [hive] yigress opened a new pull request, #3840: HIVE-26815: Backport HIVE-26758 (Allow use scratchdir for staging fin…
yigress opened a new pull request, #3840:
URL: https://github.com/apache/hive/pull/3840
### What changes were proposed in this pull request?
1. add a hive configuration hive.use.scratchdir.for.staging
2. for native table, no-mm, no-direct-insert, no-acid, change dynamic partition staging directory layout from
<dest_path>/<static_partition>/<staging_dir>/<dynamic_partition>
to
<dest_path>/<staging_dir>/<static_partition>/<dynamic_partition>
3. when hive.use.scratchdir.for.staging=true, FileSinkOperator's dirName, DynamicContext's sourcePath change from
<dest_path>/{hive.exec.stagingdir}
to
<hive.exec.scratchdir>
for example for query
insert into/overwrite table partition(year=2001, season) select...
before the change, the FileSinkOperator conf has
<table_path>/year=2001/.staging_dir/season=xxx
after the change, it has
<table_path>/.staging_dir/year=2001/season=xxx
This change allow to swap <table_path> with another path such as <hive.exec.scratchdir>, and the moveTask will move into <table_path>
### Why are the changes needed?
In the S3 blobstorage optimization, HIVE-15121 / HIVE-17620 changed interim job path to use hive.exec.scracthdir, final job to use hive.exec.stagingdir. https://issues.apache.org/jira/browse/HIVE-15215 is open whether to use scratch for staging dir for S3.
However for blobstorage where 'rename' is slow and no encryption, it can help performance to use scratchdir to staging query results and use the MoveTask to copy to blobstorage. This is especially true when there is FileMerge task.
This may also help cross-filesystem when user wants to use local cluster filesystem to staging query results and move the results to destination filesystem.
### Does this PR introduce _any_ user-facing change?
This adds a new hive configuration hive.use.scratchdir.for.staging, default false
### How was this patch tested?
Tested with patch on hive 3.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao merged pull request #3840: HIVE-26815: Backport HIVE-26758 (Allow use scratchdir for staging fin…
Posted by GitBox <gi...@apache.org>.
sunchao merged PR #3840:
URL: https://github.com/apache/hive/pull/3840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #3840: HIVE-26815: Backport HIVE-26758 (Allow use scratchdir for staging fin…
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #3840:
URL: https://github.com/apache/hive/pull/3840#issuecomment-1341615164
Merged, thanks @yigress !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org