You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:21:36 UTC
[jira] [Updated] (SPARK-8672) throws NPE when running spark sql
thrift server with session state authenticator
[ https://issues.apache.org/jira/browse/SPARK-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-8672:
--------------------------------
Labels: bulk-closed (was: )
> throws NPE when running spark sql thrift server with session state authenticator
> --------------------------------------------------------------------------------
>
> Key: SPARK-8672
> URL: https://issues.apache.org/jira/browse/SPARK-8672
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Environment: Hive 0.13.1
> Reporter: Min Zhou
> Priority: Major
> Labels: bulk-closed
>
> Here is the configuration
> {noformat}
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.security.authorization.manager</name>
> <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
> </property>
> <property>
> <name>hive.security.metastore.authorization.manager</name>
> <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
> </property>
> <property>
> <name>hive.security.authenticator.manager</name>
> <value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
> </property>
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>true</value>
> </property>
> {noformat}
> I started spark thrift server with -hiveconf hive.security.authorization.enabled=true
> when logining to beeline, connect with jdbc and create a table after that, a NPE throws. The stack is like below
> {noformat}
> FAILED: NullPointerException null org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:333)
> org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:310)
> org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:139)
> org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:310)
> org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:300)
> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:472)
> org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)
> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
> org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:939)
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:939)
> org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144)
> org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128)
> org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
> org.apache.spark.sql.SQLContext.sql(SQLContext.scala:744)
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:178)
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:415)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493)
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
> com.sun.proxy.$Proxy39.executeStatementAsync(Unknown Source)
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344)
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {noformat}
> After some deep tracing, i notice the root cause is a null username inside a SessionState object, which is passed to a SessionStateUserAuthenticator instance. The stack above shows it doesn't get the right SessionState. Instead get a SessionState from thread local.
> On the other hand, every time a user login. a new SessionState with a correct username will be created
> {noformat}
> org/apache/hadoop/hive/ql/session/SessionState.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;Ljava/lang/String;)V call by thread [pool-20-thread-3]
> org.apache.hive.service.cli.session.HiveSessionImpl.<init>(HiveSessionImpl.java:104)
> org.apache.hive.service.cli.session.HiveSessionImplwithUGI.<init>(HiveSessionImplwithUGI.java:49)
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:122)
> org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(Shim13.scala:241)
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:161)
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:265)
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:191)
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {noformat}
> However, the CREATE TABLE statement never retrieve this sessionstate, running queries upon this right instance.
> So , NPE happens
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org