You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "llai (JIRA)" <ji...@apache.org> on 2014/05/25 14:22:01 UTC
[jira] [Commented] (SPARK-1922) hql query throws "RuntimeException:
Unsupported dataType" if table has a struct field which has a column with
underscore in its name
[ https://issues.apache.org/jira/browse/SPARK-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008325#comment-14008325 ]
llai commented on SPARK-1922:
-----------------------------
My table uses avro serde. Checked Hive [LanguageManual DDL|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable] and [Avro spec|https://avro.apache.org/docs/1.7.6/spec.html#Names], underscore is valid.
> hql query throws "RuntimeException: Unsupported dataType" if table has a struct field which has a column with underscore in its name
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-1922
> URL: https://issues.apache.org/jira/browse/SPARK-1922
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.0.0
> Reporter: llai
>
> If table A has a struct field A_strct <a_col: string>, when doing an hql query like "select", "RuntimeException: Unsupported dataType" is thrown.
> Running a query wiht "sbt/sbt hive/console":
> {code}
> scala> hql("SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10").collect().foreach(x => println(x(1)))
> {code}
> Console output:
> {code}
> 14/05/25 19:50:27 INFO parse.ParseDriver: Parsing command: SELECT utc_time, pkg FROM pkg_table where year=2014 and month=1 limit 10
> 14/05/25 19:50:27 INFO parse.ParseDriver: Parse Completed
> 14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
> 14/05/25 19:50:28 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
> 14/05/25 19:50:28 INFO hive.metastore: Trying to connect to metastore with URI thrift://xxxxx
> 14/05/25 19:50:28 INFO hive.metastore: Waiting 1 seconds before next connection attempt.
> 14/05/25 19:50:29 INFO hive.metastore: Connected to metastore.
> java.lang.RuntimeException: Unsupported dataType: struct<adv_id:string,u_lat:double,u_lon:double>
> at scala.sys.package$.error(package.scala:27)
> at org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:219)
> at org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:273)
> at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
> at org.apache.spark.sql.hive.MetastoreRelation$$anonfun$8.apply(HiveMetastoreCatalog.scala:283)
> at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)