You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/06/20 03:51:00 UTC
[jira] [Resolved] (SPARK-35762) Errors while using spark-sql read
hive 3.1 orc table
[ https://issues.apache.org/jira/browse/SPARK-35762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-35762.
----------------------------------
Resolution: Duplicate
It's a duplicate of SPARK-35700
> Errors while using spark-sql read hive 3.1 orc table
> ----------------------------------------------------
>
> Key: SPARK-35762
> URL: https://issues.apache.org/jira/browse/SPARK-35762
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1, 3.1.2
> Environment: h1. **** env ****
> OS: Centos7.7 x86_64
> JDK: Oracle JDK 1.8.0_191
> Hadoop: 2.8.4
> Hive: 3.1.2
> Spark: 3.1.0
>
> A simple 3 nodes hadoop cluster.
> Hive with metastore service started.
> Reporter: laokong
> Priority: Major
> Attachments: full-stack-trace.log
>
>
> === recreate problem steps
> 1.create orc table in hive
> ```
> hive> drop table demo;
> OK
> Time taken: 0.514 seconds
> hive> create table demo(`id` varchar(20)) stored as orc;
> OK
> Time taken: 0.134 seconds
> hive> insert into table demo(id) values('111');
> hive> select * from demo;
> OK
> 111
> Time taken: 0.291 seconds, Fetched: 1 row(s)
> ```
>
> 2.copy hive-site.xml to spark/conf
> ```
> cd /root/spark-3.1.2-bin-hadoop2.7/conf
> ln -s /usr/hive/conf/hive-site.xml .
> ```
> 3.exec SQL in spark-sql
> ```
> [root@master spark-3.1.2-bin-hadoop2.7]# ./bin/spark-sql[root@master spark-3.1.2-bin-hadoop2.7]# ./bin/spark-sql21/06/14 23:32:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).21/06/14 23:32:17 WARN conf.HiveConf: HiveConf of name hive.metastore.db.type does not exist
> Spark master: local[*], Application Id: local-1623727939853spark-sql> select * from demo where id='222';
> java.lang.UnsupportedOperationException: DataType: varchar(20)java.lang.UnsupportedOperationException: DataType: varchar(20) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.getPredicateLeafType(OrcFilters.scala:150) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.getType$1(OrcFilters.scala:222) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:266) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.convertibleFiltersHelper$1(OrcFilters.scala:132) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.$anonfun$convertibleFilters$4(OrcFilters.scala:135) at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245) at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242) at scala.collection.immutable.List.flatMap(List.scala:355) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.convertibleFilters(OrcFilters.scala:134) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.createFilter(OrcFilters.scala:73) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$4(OrcFileFormat.scala:189) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$4$adapted(OrcFileFormat.scala:188) at scala.Option.foreach(Option.scala:407) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$1(OrcFileFormat.scala:188) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:169) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:503) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> ```
>
> ==== summary ====
> 1.Tested in spark-3.1.1/spark-3.1.2 report this error.
> If start spark-sql with option `-c spark.sql.orc.impl=hive` ,problem disappear
>
> 2.Tested in spark-3.0.2 without any problem.
>
> 3.spark 3.1.2 with option `-c spark.sql.orc.impl=hive` performance is slower than spark3.0.2 without this option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org