You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Alagappan Maruthappan (Jira)" <ji...@apache.org> on 2023/02/03 22:10:00 UTC
[jira] [Updated] (HIVE-27023) Add setting to disable hive session being started during Hive Cli startup
[ https://issues.apache.org/jira/browse/HIVE-27023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alagappan Maruthappan updated HIVE-27023:
-----------------------------------------
Description:
Added a setting disable Tez session being opened during the startup of Hive cli. This is particularly needed when Hive action is being launched by Oozie to perform DML only operations. In that cases before the tez session thread could complete the launcher gets completed and clears all the scratch directories. This leads to a failure with following stack trace:
{code:java}
Application application_1667416881396_24229473 failed 3 times due to AM Container for appattempt_1667416881396_24229473_000003 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not exist: hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException: File does not exist: hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2136)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2116)at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:249)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:242)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:230)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748) {code}
Also opening up of tez session in such cases is not needed.
was:
Added a setting disable Tez session being opened during the startup of Hive cli. This is particularly needed when Hive action is being launched by Oozie to perform DML only operations. In that cases before the tez session thread could complete the launcher gets completed and clears all the scratch directories. This leads to a failure with following stack trace:
{code:java}
Application application_1667416881396_24229473 failed 3 times due to AM Container for appattempt_1667416881396_24229473_000003 exited with exitCode: -1000Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not exist: hdfs://jetblue-nn1.blue.ygrid.yahoo.com:8020/tmp/megaman/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException: File does not exist: hdfs://jetblue-nn1.blue.ygrid.yahoo.com:8020/tmp/megaman/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2136)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2116)at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:249)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:242)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:230)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)For more detailed output, check the application tracking page: https://jetblue-jt1.blue.ygrid.yahoo.com:50508/applicationhistory/app/application_1667416881396_24229473 Then click on links to logs of each attempt.. Failing the application. {code}
Also opening up of tez session in such cases is not needed.
> Add setting to disable hive session being started during Hive Cli startup
> -------------------------------------------------------------------------
>
> Key: HIVE-27023
> URL: https://issues.apache.org/jira/browse/HIVE-27023
> Project: Hive
> Issue Type: Improvement
> Reporter: Alagappan Maruthappan
> Assignee: Alagappan Maruthappan
> Priority: Minor
>
> Added a setting disable Tez session being opened during the startup of Hive cli. This is particularly needed when Hive action is being launched by Oozie to perform DML only operations. In that cases before the tez session thread could complete the launcher gets completed and clears all the scratch directories. This leads to a failure with following stack trace:
> {code:java}
> Application application_1667416881396_24229473 failed 3 times due to AM Container for appattempt_1667416881396_24229473_000003 exited with exitCode: -1000
> Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not exist: hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException: File does not exist: hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2136)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2116)at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:249)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:242)at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:230)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748) {code}
> Also opening up of tez session in such cases is not needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)