You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Nikhil Gupta (Jira)" <ji...@apache.org> on 2021/10/07 17:06:00 UTC
[jira] [Created] (HIVE-25600) Compaction job creates redundant
base/delta folder within base/delta folder
Nikhil Gupta created HIVE-25600:
-----------------------------------
Summary: Compaction job creates redundant base/delta folder within base/delta folder
Key: HIVE-25600
URL: https://issues.apache.org/jira/browse/HIVE-25600
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.2, 3.1.0
Reporter: Nikhil Gupta
{noformat}
Hive table 'myntra_wms.myntra_wms_item' is corrupt. Found sub-directory 'abfs://bifrostx-hive-data@gen2hivebifros.dfs.core.windows.net/prod-data/myntra_wms.db/myntra_wms_item/part_created_on=202105/base_0004042/base_0004042' in bucket directory for partition: part_created_on=202105
at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:543)
at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:325)
at io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:254)
at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
at io.prestosql.$gen.Presto_347____20210615_143054_2.run(Unknown Source)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829);{noformat}
Why it happens:
Multiple compaction jobs for the same transactions can be triggered if the HMS gets restarted and the MR job is still in progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)