You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2014/10/29 03:12:33 UTC
[jira] [Resolved] (SPARK-3904) HQL doesn't support the
ConstantObjectInspector to pass into UDFs
[ https://issues.apache.org/jira/browse/SPARK-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-3904.
-------------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
Issue resolved by pull request 2762
[https://github.com/apache/spark/pull/2762]
> HQL doesn't support the ConstantObjectInspector to pass into UDFs
> -----------------------------------------------------------------
>
> Key: SPARK-3904
> URL: https://issues.apache.org/jira/browse/SPARK-3904
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Cheng Hao
> Assignee: Cheng Hao
> Fix For: 1.2.0
>
>
> In HQL, we convert all of the data type into normal ObjectInspectors for UDFs, most of cases it work, however, some of the UDF actually requires the input ObjectInspector to be the ConstantObjectInspector, which will cause exception.
> e.g.
> {panel}
> select named_struct("x", "str") from src limit 1
> {panel}
> It will throws exception like
> {panel}
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parsing command: select named_struct("x", "str") from src
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parse Completed
> 14/10/10 16:25:17 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src
> 14/10/10 16:25:17 INFO HiveMetaStore.audit: ugi=hcheng ip=unknown-ip-addr cmd=get_table : db=default tbl=tmp2
> 14/10/10 16:25:17 ERROR thriftserver.SparkSQLDriver: Failed in [select named_struct("x", "str") from src]
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Even arguments to NAMED_STRUCT must be a constant STRING.org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector@2f2dbcfc
> at org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct.initialize(GenericUDFNamedStruct.java:55)
> at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector$lzycompute(hiveUdfs.scala:129)
> at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector(hiveUdfs.scala:129)
> at org.apache.spark.sql.hive.HiveGenericUdf.eval(hiveUdfs.scala:158)
> at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:267)
> at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:260)
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org