You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Vipin (Jira)" <ji...@apache.org> on 2021/09/27 18:54:00 UTC
[jira] [Created] (HIVE-25565) Materialized view Rebuild issue Aws
EMR
Vipin created HIVE-25565:
----------------------------
Summary: Materialized view Rebuild issue Aws EMR
Key: HIVE-25565
URL: https://issues.apache.org/jira/browse/HIVE-25565
Project: Hive
Issue Type: Bug
Environment: ```
EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
```
Reporter: Vipin
We have Materialized views built on top of Hudi tables which are hive-sync'd.We have Materialized views built on top of Hudi tables which are hive-sync'd.
Hive uses AWS Glue for its metastore catalog. We are running into issue whenever we are trying to "**rebuild**" Hive materialized views. Please note, creation of materialized views works fine. It's only rebuild which is failing. However, it does seem the rebuild actually seems to work behind the scenes but its throws some exception causing EMR steps to fail.
Can anyone please guide us here, about any config changes that we need to do or anything. Any help will be great.
```
EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
```
The stack trace of the exception -
> FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking FailureHook. hooks: java.lang.NullPointerException at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45) at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296) at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283) at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748))
> org.apache.hadoop.hive.ql.metadata.HiveException: Error while invoking FailureHook. hooks:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> > at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:302)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> Caused by: java.lang.NullPointerException> at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> ... 19 more
> INFO : Completed executing command(queryId=hive_20210927174227_c0ccf861-e7df-4ae4-be7d-b93774d29c25); Time taken: 15.402 seconds> INFO : Concurrency mode is disabled, not creating a lock manager> Error: Error while processing statement: FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking FailureHook. hooks: java.lang.NullPointerException> at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> ) (state=08S01,code=12)>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)