You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2023/01/04 11:36:00 UTC
[jira] [Created] (HIVE-26904) QueryCompactor failed in commitCompaction if the tmp table dir is already removed
Quanlong Huang created HIVE-26904:
-------------------------------------
Summary: QueryCompactor failed in commitCompaction if the tmp table dir is already removed
Key: HIVE-26904
URL: https://issues.apache.org/jira/browse/HIVE-26904
Project: Hive
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
commitCompaction() of query-based compactions just remove the dirs of tmp tables. It should not fail the compaction if the dirs are already removed.
We've seen such a failure in Impala's test (IMPALA-11756):
{noformat}
2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695] ql.Driver: Executing command(queryId=jenkins_20230102020926_69112755-b783-4214-89e5-1c7111dfe15f): alter table partial_catalog_info_test.insert_only_partitioned partition (part=1) compact 'minor' and wait
2023-01-02T02:09:26,306 INFO [HiveServer2-Background-Pool: Thread-695] ql.Driver: Starting task [Stage-0:DDL] in serial mode
2023-01-02T02:09:26,317 INFO [HiveServer2-Background-Pool: Thread-695] exec.Task: Compaction enqueued with id 15
...
2023-01-02T02:12:55,849 ERROR [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.Worker: Caught exception while trying to compact id:15,dbname:partial_catalog_info_test,tableName:insert_only_partitioned,partName:part=1,state:^@,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:jenkins,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId: null,initiatorId: null,retryRetention0. Marking failed to avoid repeated failures
java.io.FileNotFoundException: File hdfs://localhost:20500/tmp/hive/jenkins/092b533a-81c8-4b95-88e4-9472cf6f365d/_tmp_space.db/62ec04fb-e2d2-4a99-a454-ae709a3cccfe does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1275) ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249) ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194) ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190) ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208) ~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144) ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:2302) ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2299) ~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor$Util.cleanupEmptyDir(QueryCompactor.java:261) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.commitCompaction(MmMinorQueryCompactor.java:72) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:146) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.runCompaction(MmMinorQueryCompactor.java:63) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:435) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:115) ~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
2023-01-02T02:12:55,858 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.Worker: Deleting result directories created by the compactor:2023-01-02T02:12:55,858 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.Worker: hdfs://localhost:20500/test-warehouse/managed/partial_catalog_info_test.db/insert_only_partitioned/part=1/delta_0000001_0000003_v0001827
2023-01-02T02:12:55,859 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.Worker: hdfs://localhost:20500/test-warehouse/managed/partial_catalog_info_test.db/insert_only_partitioned/part=1/delete_delta_0000001_0000003_v0001827
2023-01-02T02:12:55,859 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.CompactionHeartbeatService: Stopping heartbeat task for TXN 1827
2023-01-02T02:12:55,859 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.CompactionHeartbeatService$CompactionHeartbeater: Shutting down compaction txn heartbeater instance.
2023-01-02T02:12:55,859 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.CompactionHeartbeatService$CompactionHeartbeater: Compaction txn heartbeater instance is successfully stopped.
2023-01-02T02:12:55,872 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48] compactor.Worker: Worker thread finished one loop.
2023-01-02T02:12:55,872 INFO [impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor] compactor.Worker: Processing compaction request null
2023-01-02T02:12:56,400 INFO [HiveServer2-Background-Pool: Thread-695] exec.Task: Compaction with id 15 finished with status: failed {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)