You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Karen Coppage (Jira)" <ji...@apache.org> on 2020/10/15 08:17:00 UTC
[jira] [Resolved] (HIVE-24235) Drop and recreate table during MR
compaction leaves behind base/delta directory
[ https://issues.apache.org/jira/browse/HIVE-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karen Coppage resolved HIVE-24235.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master branch. Thanks [~pvary] for the review!
> Drop and recreate table during MR compaction leaves behind base/delta directory
> -------------------------------------------------------------------------------
>
> Key: HIVE-24235
> URL: https://issues.apache.org/jira/browse/HIVE-24235
> Project: Hive
> Issue Type: Bug
> Reporter: Karen Coppage
> Assignee: Karen Coppage
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If a table is dropped and recreated during MR compaction, the table directory and a base (or delta, if minor compaction) directory could be created, with or without data, while the table "does not exist".
> E.g.
> {code:java}
> create table c (i int) stored as orc tblproperties ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> insert into c values (9);
> insert into c values (9);
> alter table c compact 'major';
> While compaction job is running: {
> drop table c;
> create table c (i int) stored as orc tblproperties ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> }
> {code}
> The table directory should be empty, but table directory could look like this after the job is finished:
> {code:java}
> Oct 6 14:23 c/base_0000002_v0000101/._orc_acid_version.crc
> Oct 6 14:23 c/base_0000002_v0000101/.bucket_00000.crc
> Oct 6 14:23 c/base_0000002_v0000101/_orc_acid_version
> Oct 6 14:23 c/base_0000002_v0000101/bucket_00000
> {code}
> or perhaps just:
> {code:java}
> Oct 6 14:23 c/base_0000002_v0000101/._orc_acid_version.crc
> Oct 6 14:23 c/base_0000002_v0000101/_orc_acid_version
> {code}
> Insert another row and you have:
> {code:java}
> Oct 6 14:33 base_0000002_v0000101/
> Oct 6 14:33 base_0000002_v0000101/._orc_acid_version.crc
> Oct 6 14:33 base_0000002_v0000101/.bucket_00000.crc
> Oct 6 14:33 base_0000002_v0000101/_orc_acid_version
> Oct 6 14:33 base_0000002_v0000101/bucket_00000
> Oct 6 14:35 delta_0000001_0000001_0000/._orc_acid_version.crc
> Oct 6 14:35 delta_0000001_0000001_0000/.bucket_00000_0.crc
> Oct 6 14:35 delta_0000001_0000001_0000/_orc_acid_version
> Oct 6 14:35 delta_0000001_0000001_0000/bucket_00000_0
> {code}
> Selecting from the table will result in this error because the highest valid writeId for this table is 1:
> {code:java}
> thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Unable to get the next row set
> at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:482) ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> ...
> Caused by: java.io.IOException: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: Not enough history available for (1,x). Oldest available base: .../warehouse/b/base_0000004_v0000092
> {code}
> Solution: Resolve the table again after compaction is finished; compare the id with the table id from when compaction began. If the ids do not match, abort the compaction's transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)