You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2022/07/05 02:41:00 UTC
[jira] [Updated] (SPARK-39671) insert overwrite table java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition .This problem occurred when we installed Apache Spark3.0.1-hadoop3.0 in CDH6.1.1
[ https://issues.apache.org/jira/browse/SPARK-39671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-39671:
---------------------------------
Component/s: SQL
(was: Pandas API on Spark)
> insert overwrite table java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition .This problem occurred when we installed Apache Spark3.0.1-hadoop3.0 in CDH6.1.1
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-39671
> URL: https://issues.apache.org/jira/browse/SPARK-39671
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Reporter: xin
> Priority: Major
>
> use spark-thrifter run this sql insert overwrite table xx.xx partition(dt=2022-06-30) select * from xxx.xxx; The SQL execution environment is cdh 6.1.1 hive version 2.1.1
>
>
> raise OperationalError(response) pyhive.exc.OperationalError: TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Error running query: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, boolean, boolean, boolean, boolean, boolean):25:24', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:321', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:202', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:278', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkOperation$$super$run:SparkExecuteStatementOperation.scala:46', 'org.apache.spark.sql.hive.thriftserver.SparkOperation:$anonfun$run$1:SparkOperation.scala:44', 'scala.runtime.java8.JFunction0$mcV$sp:apply:JFunction0$mcV$sp.java:23', 'org.apache.spark.sql.hive.thriftserver.SparkOperation:withLocalProperties:SparkOperation.scala:78', 'org.apache.spark.sql.hive.thriftserver.SparkOperation:withLocalProperties$:SparkOperation.scala:62', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:withLocalProperties:SparkExecuteStatementOperation.scala:46', 'org.apache.spark.sql.hive.thriftserver.SparkOperation:run:SparkOperation.scala:44', 'org.apache.spark.sql.hive.thriftserver.SparkOperation:run$:SparkOperation.scala:42', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:run:SparkExecuteStatementOperation.scala:46', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:484', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:460', 'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:280', 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:439', 'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437', 'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:53', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:310', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', 'java.lang.Thread:run:Thread.java:748', '*java.lang.NoSuchMethodException:org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, boolean, boolean, boolean, boolean, boolean):63:38', 'java.lang.Class:getMethod:Class.java:1786', 'org.apache.spark.sql.hive.client.Shim:findMethod:HiveShim.scala:177', 'org.apache.spark.sql.hive.client.Shim_v2_1:loadPartitionMethod$lzycompute:HiveShim.scala:1151', 'org.apache.spark.sql.hive.client.Shim_v2_1:loadPartitionMethod:HiveShim.scala:1139', 'org.apache.spark.sql.hive.client.Shim_v2_1:loadPartition:HiveShim.scala:1201', 'org.apache.spark.sql.hive.client.HiveClientImpl:$anonfun$loadPartition$1:HiveClientImpl.scala:872', 'scala.runtime.java8.JFunction0$mcV$sp:apply:JFunction0$mcV$sp.java:23', 'org.apache.spark.sql.hive.client.HiveClientImpl:$anonfun$withHiveState$1:HiveClientImpl.scala:294', 'org.apache.spark.sql.hive.client.HiveClientImpl:liftedTree1$1:HiveClientImpl.scala:227', 'org.apache.spark.sql.hive.client.HiveClientImpl:retryLocked:HiveClientImpl.scala:226', 'org.apache.spark.sql.hive.client.HiveClientImpl:withHiveState:HiveClientImpl.scala:276', 'org.apache.spark.sql.hive.client.HiveClientImpl:loadPartition:HiveClientImpl.scala:862', 'org.apache.spark.sql.hive.HiveExternalCatalog:$anonfun$loadPartition$1:HiveExternalCatalog.scala:915', 'scala.runtime.java8.JFunction0$mcV$sp:apply:JFunction0$mcV$sp.java:23', 'org.apache.spark.sql.hive.HiveExternalCatalog:withClient:HiveExternalCatalog.scala:103', 'org.apache.spark.sql.hive.HiveExternalCatalog:loadPartition:HiveExternalCatalog.scala:894', 'org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener:loadPartition:ExternalCatalogWithListener.scala:179', 'org.apache.spark.sql.hive.execution.InsertIntoHiveTable:processInsert:InsertIntoHiveTable.scala:316', 'org.apache.spark.sql.hive.execution.InsertIntoHiveTable:run:InsertIntoHiveTable.scala:102', 'org.apache.spark.sql.execution.command.DataWritingCommandExec:sideEffectResult$lzycompute:commands.scala:108', 'org.apache.spark.sql.execution.command.DataWritingCommandExec:sideEffectResult:commands.scala:106', 'org.apache.spark.sql.execution.command.DataWritingCommandExec:executeCollect:commands.scala:120', 'org.apache.spark.sql.Dataset:$anonfun$logicalPlan$1:Dataset.scala:229', 'org.apache.spark.sql.Dataset:$anonfun$withAction$1:Dataset.scala:3618', 'org.apache.spark.sql.execution.SQLExecution$:$anonfun$withNewExecutionId$5:SQLExecution.scala:100', 'org.apache.spark.sql.execution.SQLExecution$:withSQLConfPropagated:SQLExecution.scala:160', 'org.apache.spark.sql.execution.SQLExecution$:$anonfun$withNewExecutionId$1:SQLExecution.scala:87', 'org.apache.spark.sql.SparkSession:withActive:SparkSession.scala:764', 'org.apache.spark.sql.execution.SQLExecution$:withNewExecutionId:SQLExecution.scala:64', 'org.apache.spark.sql.Dataset:withAction:Dataset.scala:3616', 'org.apache.spark.sql.Dataset:<init>:Dataset.scala:229', 'org.apache.spark.sql.Dataset$:$anonfun$ofRows$2:Dataset.scala:100', 'org.apache.spark.sql.SparkSession:withActive:SparkSession.scala:764', 'org.apache.spark.sql.Dataset$:ofRows:Dataset.scala:97', 'org.apache.spark.sql.SparkSession:$anonfun$sql$1:SparkSession.scala:607', 'org.apache.spark.sql.SparkSession:withActive:SparkSession.scala:764', 'org.apache.spark.sql.SparkSession:sql:SparkSession.scala:602', 'org.apache.spark.sql.SQLContext:sql:SQLContext.scala:650', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:280'], sqlState=None, errorCode=0, errorMessage='Error running query: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, boolean, boolean, boolean, boolean, boolean)'), operationHandle=None)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org