You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sruthi M (Jira)" <ji...@apache.org> on 2022/12/15 05:57:00 UTC
[jira] [Comment Edited] (HIVE-26815) Backport HIVE-26758 (Allow use scratchdir for staging final job)
[ https://issues.apache.org/jira/browse/HIVE-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647861#comment-17647861 ]
Sruthi M edited comment on HIVE-26815 at 12/15/22 5:56 AM:
-----------------------------------------------------------
Hi [~yigress], this patch doesn't seem to work as expected.
i.e, The staging files seem to be getting created in the table location itself (abfs in the below case), and not using scratch dir (which is hdfs://mycluster/tmp/hive) even after enabling the parameter 'hive.use.scratchdir.for.staging'.
{noformat}
0: jdbc:hive2://xxxxxx> set hive.use.scratchdir.for.staging;
+---------------------------------------+
| set |
+---------------------------------------+
| hive.use.scratchdir.for.staging=true |
+---------------------------------------+
{noformat}
{noformat}
2022-12-15T04:07:24,535 INFO [load-dynamic-partitions-77] common.FileUtils: Creating directory if it doesn't exist: abfs://xxxxx@zzzzz/hive/warehouse/managed/tpcds_orc.db/teste_store_sales_fourthrun/ss_sold_date_sk=2452172
2022-12-15T04:07:24,578 INFO [load-dynamic-partitions-75] common.FileUtils: Creating directory if it doesn't exist: abfs://xxxxx@zzzzz/hive/warehouse/managed/tpcds_orc.db/teste_store_sales_fourthrun/ss_sold_date_sk=2451407
2022-12-15T04:07:24,620 INFO [load-dynamic-partitions-79] metadata.Hive: Moving bucket abfs://xxxxx@zzzzz/hive/warehouse/managed/tpcds_orc.db/teste_store_sales_fourthrun/.hive-staging_hive_2022-12-15_04-03-23_618_7688646701933279020-2/-ext-10000/ss_sold_date_sk=2452481/000006_0/base_0000001/bucket_00006 to abfs://xxxxx@zzzzz/hive/warehouse/managed/tpcds_orc.db/teste_store_sales{noformat}
Creating the staging directory on hdfs is not working as shown above. Can you please check this and let me know what more needs to be configured for achieving the same?
was (Author: sruthim-official):
Hi [~yigress], this patch doesn't seem to work as expected.
i.e, The staging files seem to be getting created in the table location itself (abfs in the below case), and not using scratch dir (which is hdfs://mycluster/tmp/hive) even after enabling the parameter 'hive.use.scratchdir.for.staging'. See the attachments.
Creating the staging directory on hdfs is not working as shown in the attachments. Can you please check this and let me know what more needs to be configured for achieving the same?
> Backport HIVE-26758 (Allow use scratchdir for staging final job)
> ----------------------------------------------------------------
>
> Key: HIVE-26815
> URL: https://issues.apache.org/jira/browse/HIVE-26815
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Affects Versions: 3.1.3
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-26758 add an option to allow choose set final job staging with hive.exec.scratchdir. This is to backport this into 3.2.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)