You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Fernando Antunes <fs...@gmail.com> on 2020/02/28 19:20:48 UTC
LLAP - Hive on Tez - ERROR on SQL query
Hi.
I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works.
When I run a SQL query like
select count(*) from database group by column;
I've got the following error:
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
The error only happens when I use LLAP, using HiverServer2 works.
Anyone knows how to fix this ?
More details
0) I’ve searched the Internet and found others with the same error but without answer
http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cDB7P194MB0380535868265DF6862B849E93650@DB7P194MB0380.EURP194.PROD.OUTLOOK.COM%3e <http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3CDB7P194MB0380535868265DF6862B849E93650@DB7P194MB0380.EURP194.PROD.OUTLOOK.COM%3E>
https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597 <https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597>
1) Application log form YARN
2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
org.apache.tez.dag.api.TezUncheckedException: java.lang.reflect.InvocationTargetException
at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215)
at org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184)
at org.apache.tez.dag.app.TaskCommunicatorManager.<init>(TaskCommunicatorManager.java:152)
at org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213)
... 11 more
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
... 16 more
2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2) From the LlapTaskCommunicator.java file I found :
public LlapTaskCommunicator(
TaskCommunicatorContext taskCommunicatorContext) {
super(taskCommunicatorContext);
Credentials credentials = taskCommunicatorContext.getAMCredentials();
if (credentials != null) {
@SuppressWarnings("unchecked")
Token<LlapTokenIdentifier> llapToken =
(Token<LlapTokenIdentifier>)credentials.getToken(LlapTokenIdentifier.KIND_NAME);
this.token = llapToken;
} else {
this.token = null;
}
if (LOG.isInfoEnabled()) {
LOG.info("Task communicator with a token " + token);
}
Preconditions.checkState((token != null) == UserGroupInformation.isSecurityEnabled()); <=====. Line 165
3) From my application log I could see that token variable is not null
Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN; LLAP_TOKEN owner=hive/xxxxx.aaaaa.com@XXXXX.AAAAA.COM, renewer=hive, realUser=, issueDate=1582827878248, maxDate=1584037478248, sequenceNumber=59, masterKeyId=184, cluster hive_llap0, app ID , signing false)
3) Reading Preconditions.java the checkState method
* Ensures the truth of an expression involving the state of the calling instance, but not
* involving any parameters to the calling method.
*
* @param expression a boolean expression
* @throws IllegalStateException if {@code expression} is false
* @see Verify#verify Verify.verify()
*/
public static void checkState(boolean expression) {
if (!expression) {
throw new IllegalStateException();
}
}
4) from UseGroupInformation.java : the isSecurityEnabled method. ensureInitialized does not change useKerberos variable.
public static boolean isSecurityEnabled() {
ensureInitialized();
return useKerberos;
}
5) useKerberos is true, because my core-site-.xml file has kerberos value set to the HADOOP_SECURITY_AUTHENTICATION property
private static synchronized void initialize(Configuration conf, boolean skipRulesSetting) {
String value = conf.get(HADOOP_SECURITY_AUTHENTICATION);
if (value == null || "simple".equals(value)) {
useKerberos = false;
} else if ("kerberos".equals(value)) {
useKerberos = true;
} else {
throw new IllegalArgumentException("Invalid attribute value for " +
HADOOP_SECURITY_AUTHENTICATION +
" of " + value);
}
6) Form CommonConfigurationKeys.java, HADOOP_SECURITY_AUTHENTICATION points to hadoop.security.authentication
public static final String HADOOP_SECURITY_AUTHENTICATION =
"hadoop.security.authentication";
RE: LLAP - Hive on Tez - ERROR on SQL query
Posted by Aaron Grubb <Aa...@clearpier.com>.
I experienced weird classpath issues when deploying LLAP on a from-scratch cluster. Things like, it wasn’t able to find hive-default.xml and had no default configuration provider, so I had to include a hive-default.xml with every possible setting inside the Yarnfile package to get it to run. This doesn’t necessarily apply to HDP but my suggestion would be to trace which classes/configurations are being loaded as much as possible and verify they line up with what you expected.
From: Fernando Antunes <fs...@gmail.com>
Sent: Friday, February 28, 2020 2:21 PM
To: user@hive.apache.org
Subject: LLAP - Hive on Tez - ERROR on SQL query
Hi.
I’m trying to make LLAP on HDP 3.1.4 with Hive 3.1.0 and Kerberos enabled works.
When I run a SQL query like
select count(*) from database group by column;
I've got the following error:
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
The error only happens when I use LLAP, using HiverServer2 works.
Anyone knows how to fix this ?
More details
0) I’ve searched the Internet and found others with the same error but without answer
http://mail-archives.apache.org/mod_mbox/tez-user/201910.mbox/%3cDB7P194MB0380535868265DF6862B849E93650@DB7P194MB0380.EURP194.PROD.OUTLOOK.COM%3e
https://community.cloudera.com/t5/Support-Questions/Hive-Tez-with-llap-query-fail/m-p/270597
1) Application log form YARN
2020-02-27 18:24:43,533 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
org.apache.tez.dag.api.TezUncheckedException: java.lang.reflect.InvocationTargetException
at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:215)
at org.apache.tez.dag.app.TaskCommunicatorManager.createTaskCommunicator(TaskCommunicatorManager.java:184)
at org.apache.tez.dag.app.TaskCommunicatorManager.<init>(TaskCommunicatorManager.java:152)
at org.apache.tez.dag.app.DAGAppMaster.createTaskCommunicatorManager(DAGAppMaster.java:1163)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:536)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tez.dag.app.TaskCommunicatorManager.createCustomTaskCommunicator(TaskCommunicatorManager.java:213)
... 11 more
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.<init>(LlapTaskCommunicator.java:165)
... 16 more
2020-02-27 18:24:43,535 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2) From the LlapTaskCommunicator.java file I found :
public LlapTaskCommunicator(
TaskCommunicatorContext taskCommunicatorContext) {
super(taskCommunicatorContext);
Credentials credentials = taskCommunicatorContext.getAMCredentials();
if (credentials != null) {
@SuppressWarnings("unchecked")
Token<LlapTokenIdentifier> llapToken =
(Token<LlapTokenIdentifier>)credentials.getToken(LlapTokenIdentifier.KIND_NAME);
this.token = llapToken;
} else {
this.token = null;
}
if (LOG.isInfoEnabled()) {
LOG.info<http://LOG.info>("Task communicator with a token " + token);
}
Preconditions.checkState((token != null) == UserGroupInformation.isSecurityEnabled()); <=====. Line 165
3) From my application log I could see that token variable is not null
Task communicator with a token Kind: LLAP_TOKEN, Service: , Ident: (LLAP_TOKEN; LLAP_TOKEN owner=hive/xxxxx.aaaaa.com@XXXXX.AAAAA.COM<ma...@XXXXX.AAAAA.COM>, renewer=hive, realUser=, issueDate=1582827878248, maxDate=1584037478248, sequenceNumber=59, masterKeyId=184, cluster hive_llap0, app ID , signing false)
3) Reading Preconditions.java the checkState method
* Ensures the truth of an expression involving the state of the calling instance, but not
* involving any parameters to the calling method.
*
* @param expression a boolean expression
* @throws IllegalStateException if {@code expression} is false
* @see Verify#verify Verify.verify()
*/
public static void checkState(boolean expression) {
if (!expression) {
throw new IllegalStateException();
}
}
4) from UseGroupInformation.java : the isSecurityEnabled method. ensureInitialized does not change useKerberos variable.
public static boolean isSecurityEnabled() {
ensureInitialized();
return useKerberos;
}
5) useKerberos is true, because my core-site-.xml file has kerberos value set to the HADOOP_SECURITY_AUTHENTICATION property
private static synchronized void initialize(Configuration conf, boolean skipRulesSetting) {
String value = conf.get(HADOOP_SECURITY_AUTHENTICATION);
if (value == null || "simple".equals(value)) {
useKerberos = false;
} else if ("kerberos".equals(value)) {
useKerberos = true;
} else {
throw new IllegalArgumentException("Invalid attribute value for " +
HADOOP_SECURITY_AUTHENTICATION +
" of " + value);
}
6) Form CommonConfigurationKeys.java, HADOOP_SECURITY_AUTHENTICATION points to hadoop.security.authentication
public static final String HADOOP_SECURITY_AUTHENTICATION =
"hadoop.security.authentication";