You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2015/03/21 22:47:38 UTC
[jira] [Commented] (SPARK-6409) It is not necessary that avoid old
inteface of hive that will make some UDAF can work.
[ https://issues.apache.org/jira/browse/SPARK-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14373040#comment-14373040 ]
Michael Armbrust commented on SPARK-6409:
-----------------------------------------
Reasonable. Want to open a PR?
> It is not necessary that avoid old inteface of hive that will make some UDAF can work.
> --------------------------------------------------------------------------------------
>
> Key: SPARK-6409
> URL: https://issues.apache.org/jira/browse/SPARK-6409
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 1.2.1
> Reporter: DoingDone9
>
> I run SQL like that
> {code}
> CREATE TEMPORARY FUNCTION test_avg AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage';
>
> SELECT
> test_avg(1),
> test_avg(substr(value,5))
> FROM src;
> {code}
> then i get a exception
> {code}
> 15/03/19 09:36:45 ERROR CliDriver: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 6, HPC-3): java.lang.ClassCastException: org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage$AverageAggregationBuffer cannot be cast to org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator$AbstractAggregationBuffer
> at org.apache.spark.sql.hive.HiveUdafFunction.<init>(hiveUdfs.scala:369)
> at org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:214)
> at org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:188)
> {code}
> i find that GenericUDAFAverage used a deprecated interface AggregationBuffer that has been instead by AbstractAggregationBuffer. and spark avoid the old interface AggregationBuffer , so GenericUDAFAverage can not work.I think it is not necessary.
> code in spark
> {code}
> // Cast required to avoid type inference selecting a deprecated Hive API.
> private val buffer =
> function.getNewAggregationBuffer.asInstanceOf[GenericUDAFEvaluator.AbstractAggregationBuffer]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org