You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/07/06 21:53:00 UTC
[jira] [Work logged] (HIVE-26374) Query based compaction fails for tables with CDT and columns with Reserved Keywords
[ https://issues.apache.org/jira/browse/HIVE-26374?focusedWorklogId=788418&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-788418 ]
ASF GitHub Bot logged work on HIVE-26374:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jul/22 21:52
Start Date: 06/Jul/22 21:52
Worklog Time Spent: 10m
Work Description: cravani opened a new pull request, #3419:
URL: https://github.com/apache/hive/pull/3419
…mns with Reserved Keywords
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Query based compaction fails on Tables having complex data types with reserved keywords for columns. The compaction fails while creating a temporary table as it does not quote the columns correctly.
The changes intends to align the code used in [DDLPlanUtils.java](https://github.com/apache/hive/blob/53009126f6fe7ccf24cf052fd6c156542f38b19d/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLPlanUtils.java#L824)
### Why are the changes needed?
This fixes the current bug with Query based compaction
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Currently tested manually, I plan to add Unit test based on CI run.
Issue Time Tracking
-------------------
Worklog Id: (was: 788418)
Remaining Estimate: 0h
Time Spent: 10m
> Query based compaction fails for tables with CDT and columns with Reserved Keywords
> -----------------------------------------------------------------------------------
>
> Key: HIVE-26374
> URL: https://issues.apache.org/jira/browse/HIVE-26374
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
> Affects Versions: 3.1.3, 4.0.0-alpha-1
> Reporter: Chiran Ravani
> Assignee: Chiran Ravani
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Query based compaction fails on Tables having complex data types with reserved keywords for columns. The compaction fails while creating a temporary table as it does not quote the columns correctly.
>
> Below are the steps to reproduce the issue.
> {code:java}
> create table complex_dt_compact2(col1 array<struct<arr_col1:int, `timestamp`:string>>);
> insert into complex_dt_compact2 SELECT ARRAY(NAMED_STRUCT('arr_col1',1,'timestamp','2022-07-05 21:51:20.371'));
> insert into complex_dt_compact2 SELECT ARRAY(NAMED_STRUCT('arr_col1',2,'timestamp','2022-07-05 21:51:20.371'));
> alter table complex_dt_compact2 compact 'major' and wait; {code}
> Error:
> {code:java}
> 2022-07-05T22:15:47.710Z hiveserver2-0.hiveserver2-service.compute-1657056457-xkcx.svc.cluster.local hiveserver2 1 dbb4011d-c788-4b99-a31d-06bb6dd7182e [mdc@18060 class="compactor.Worker" level="ERROR" thread="hiveserver2-0.hiveserver2-service.compute-1657056457-xkcx.svc.cluster.local-64_executor"] Caught exception while trying to compact id:3,dbname:default,tableName:complex_dt_compact2,partName:null,state:,type:MAJOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId: null,initiatorId: null,retryRetention0. Marking failed to avoid repeated failures
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run CREATE temporary external table default_tmp_compactor_complex_dt_compact2_1657059347578(`operation` int, `originalTransaction` bigint, `bucket` int, `rowId` bigint, `currentTransaction` bigint, `row` struct<`col1` :array<struct<arr_col1:int,timestamp:string>>>) stored as orc LOCATION 's3a://obfuscated/clusters/obfuscated/obfuscated/warehouse/tablespace/managed/hive/complex_dt_compact2/base_0000003_v0000038' TBLPROPERTIES ('compactiontable'='true', 'transactional'='false')
> at org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:120)
> at org.apache.hadoop.hive.ql.txn.compactor.MajorQueryCompactor.runCompaction(MajorQueryCompactor.java:63)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:517)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run CREATE temporary external table default_tmp_compactor_complex_dt_compact2_1657059347578(`operation` int, `originalTransaction` bigint, `bucket` int, `rowId` bigint, `currentTransaction` bigint, `row` struct<`col1` :array<struct<arr_col1:int,timestamp:string>>>) stored as orc LOCATION 's3a://obfuscated/clusters/obfuscated/obfuscated/warehouse/tablespace/managed/hive/complex_dt_compact2/base_0000003_v0000038' TBLPROPERTIES ('compactiontable'='true', 'transactional'='false')
> at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:73)
> at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:50)
> at org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:113)
> ... 7 more
> Caused by: (responseCode = 40000, errorMessage = FAILED: ParseException line 1:241 cannot recognize input near 'timestamp' ':' 'string' in column specification, SQLState = 42000, exception = line 1:241 cannot recognize input near 'timestamp' ':' 'string' in column specification)
> at org.apache.hadoop.hive.ql.DriverUtils.createProcessorException(DriverUtils.java:143)
> at org.apache.hadoop.hive.ql.Compiler.handleException(Compiler.java:466)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:122)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:197)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:636)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:694)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:526)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:515)
> at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
> ... 9 more{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)