You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2014/10/14 16:42:33 UTC
[jira] [Commented] (SPARK-3904) HQL doesn't support the
ConstantObjectInspector to pass into UDFs
[ https://issues.apache.org/jira/browse/SPARK-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171011#comment-14171011 ]
Apache Spark commented on SPARK-3904:
-------------------------------------
User 'chenghao-intel' has created a pull request for this issue:
https://github.com/apache/spark/pull/2762
> HQL doesn't support the ConstantObjectInspector to pass into UDFs
> -----------------------------------------------------------------
>
> Key: SPARK-3904
> URL: https://issues.apache.org/jira/browse/SPARK-3904
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Cheng Hao
> Assignee: Cheng Hao
>
> In HQL, we convert all of the data type into normal ObjectInspectors for UDFs, most of cases it work, however, some of the UDF actually requires the input ObjectInspector to be the ConstantObjectInspector, which will cause exception.
> e.g.
> {panel}
> select named_struct("x", "str") from src limit 1
> {panel}
> It will throws exception like
> {panel}
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parsing command: select named_struct("x", "str") from src
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parse Completed
> 14/10/10 16:25:17 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src
> 14/10/10 16:25:17 INFO HiveMetaStore.audit: ugi=hcheng ip=unknown-ip-addr cmd=get_table : db=default tbl=tmp2
> 14/10/10 16:25:17 ERROR thriftserver.SparkSQLDriver: Failed in [select named_struct("x", "str") from src]
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Even arguments to NAMED_STRUCT must be a constant STRING.org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector@2f2dbcfc
> at org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct.initialize(GenericUDFNamedStruct.java:55)
> at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector$lzycompute(hiveUdfs.scala:129)
> at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector(hiveUdfs.scala:129)
> at org.apache.spark.sql.hive.HiveGenericUdf.eval(hiveUdfs.scala:158)
> at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:267)
> at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:260)
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org