You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tanvi Penumudy (Jira)" <ji...@apache.org> on 2024/06/14 07:27:00 UTC
[jira] [Comment Edited] (HDDS-10970) FileNotFoundException encountered while running hive-write in long-running setup
[ https://issues.apache.org/jira/browse/HDDS-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854955#comment-17854955 ]
Tanvi Penumudy edited comment on HDDS-10970 at 6/14/24 7:26 AM:
----------------------------------------------------------------
From the OM audits, the DELETE_KEY operation over volume: hivewritevol1717048266, bucket: hivebucket1717048272, key: hive_write/vectortab/delta_0000884_0000884_0000 appears to be successful on *2024-06-02 20:40:56* by user: hive/vc0117.<domain> with no re-creation following that.
The error came up around *2024-06-03 03:40:41* when the deleted key was being sourced.
From the OM audits, when we filter-out all operations by this user: hive/vc0117.<domain>, the results show multiple consecutive DELETE_KEY operations during and prior to the timestamp *2024-06-02 20:40.*
They all appear to be sequential deletes under the hive_write/vectortab/ directory, including our missing key suffix: delta_0000884_0000884_0000:
{code:java}
[92] delta_0000892_0000892_0000
[91] delete_delta_0000891_0000891_0000
[90] delta_0000890_0000890_0000
[89] delete_delta_0000889_0000889_0000
[88] delta_0000888_0000888_0000
[87] delete_delta_0000887_0000887_0000
[86] delta_0000886_0000886_0000
[85] delete_delta_0000885_0000885_0000
[84] delta_0000884_0000884_0000 (our missing key)
.......[so on].......
{code}
From the order of deletes, it appears that a delete query from vectortab (with specific conditions) was run at the time, causing the deletion of the directory/key, which was later referenced in another query (resulting in the error).
I believe it is safe to assume that this does not appear to be a product bug. Resolving the ticket for now, we may reopen the ticket as needed, thank you!
was (Author: JIRAUSER285056):
From the OM audits, the DELETE_KEY operation over volume: hivewritevol1717048266, bucket: hivebucket1717048272, key: hive_write/vectortab/delta_0000884_0000884_0000 appears to be successful on *2024-06-02 20:40:56* by user: hive/vc0117.<domain> with no re-creation following that.
The error came up around *2024-06-03 03:40:41* when the deleted key was being sourced.
From the OM audits, when we filter-out all operations by this user: hive/vc0117.<domain>, the results show multiple consecutive DELETE_KEY operations during and prior to the timestamp *2024-06-02 20:40.*
They all appear to be sequential deletes under the hive_write/vectortab/ directory, including our missing key suffix: delta_0000884_0000884_0000:
{code:java}
[92] delta_0000892_0000892_0000
[91] delete_delta_0000891_0000891_0000
[90] delta_0000890_0000890_0000
[89] delete_delta_0000889_0000889_0000
[88] delta_0000888_0000888_0000
[87] delete_delta_0000887_0000887_0000
[86] delta_0000886_0000886_0000
[85] delete_delta_0000885_0000885_0000
[84] delta_0000884_0000884_0000 (our missing key)
.......[so on].......
{code}
From the order of deletes, it appears that a delete query from vectortab (with specific conditions) was run at the time, causing in the deletion of the directory/key, which was later referenced in another query (resulting in the error).
I believe it is safe to assume that this does not appear to be a product bug. Resolving the ticket for now, we may reopen the ticket as needed, thank you!
> FileNotFoundException encountered while running hive-write in long-running setup
> --------------------------------------------------------------------------------
>
> Key: HDDS-10970
> URL: https://issues.apache.org/jira/browse/HDDS-10970
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Jyotirmoy Sinha
> Assignee: Tanvi Penumudy
> Priority: Major
>
> FileNotFoundException encountered while running hive-write in long-running setup
> Error stacktrace -
> {code:java}
> E ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1717317065380_0872_1_00, diagnostics=[Vertex vertex_1717317065380_0872_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: vectortab initializer failed, vertex=vertex_1717317065380_0872_1_00 [Map 1], java.lang.RuntimeException: ORC split generation failed with exception: java.io.FileNotFoundException: Unable to get file status: volume: hivewritevol1717048266 bucket: hivebucket1717048272 key: hive_write/vectortab/delta_0000884_0000884_0000
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1853)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1940)
> E at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:543)
> E at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:851)
> E at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:289)
> E at org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:203)
> E at java.security.AccessController.doPrivileged(Native Method)
> E at javax.security.auth.Subject.doAs(Subject.java:422)
> E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
> E at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:196)
> E at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:177)
> E at org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:171)
> E at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> E at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
> E at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
> E at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
> E at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> E at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> E at java.lang.Thread.run(Thread.java:748)
> E Caused by: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: Unable to get file status: volume: hivewritevol1717048266 bucket: hivebucket1717048272 key: hive_write/vectortab/delta_0000884_0000884_0000
> E at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> E at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1785)
> E ... 18 more
> E Caused by: java.io.FileNotFoundException: Unable to get file status: volume: hivewritevol1717048266 bucket: hivebucket1717048272 key: hive_write/vectortab/delta_0000884_0000884_0000
> E at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> E at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1785)
> E ... 18 more
> E Caused by: java.io.FileNotFoundException: Unable to get file status: volume: hivewritevol1717048266 bucket: hivebucket1717048272 key: hive_write/vectortab/delta_0000884_0000884_0000
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.listStatus(BasicRootedOzoneClientAdapterImpl.java:929)
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.listFileStatus(BasicRootedOzoneFileSystem.java:1250)
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.access$300(BasicRootedOzoneFileSystem.java:102)
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem$OzoneFileStatusIterator.<init>(BasicRootedOzoneFileSystem.java:1165)
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem$OzoneFileStatusIterator.<init>(BasicRootedOzoneFileSystem.java:1147)
> E at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.listStatusIterator(BasicRootedOzoneFileSystem.java:1139)
> E at org.apache.hadoop.hive.common.FileUtils.listStatusIterator(FileUtils.java:1344)
> E at org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1525)
> E at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1324)
> E at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1292)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.getAcidState(OrcInputFormat.java:1256)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:1274)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.lambda$call$0(OrcInputFormat.java:1245)
> E at java.security.AccessController.doPrivileged(Native Method)
> E at javax.security.auth.Subject.doAs(Subject.java:422)
> E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:1245)
> E at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:1210)
> E at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> E at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> E at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> E ... 3 more
> E ] {code}
> Error observed after 93 hours of execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org