You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hovell (JIRA)" <ji...@apache.org> on 2016/11/28 22:59:58 UTC
[jira] [Comment Edited] (SPARK-18527) UDAFPercentile (bigint,
array) needs explicity cast to double
[ https://issues.apache.org/jira/browse/SPARK-18527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703414#comment-15703414 ]
Herman van Hovell edited comment on SPARK-18527 at 11/28/16 10:59 PM:
----------------------------------------------------------------------
This has been fixed by merging the Percentile UDAF into branch-2.1
was (Author: hvanhovell):
This has been fixes by merging the Percentile UDAF into branch-2.1
> UDAFPercentile (bigint, array<double>) needs explicity cast to double
> ---------------------------------------------------------------------
>
> Key: SPARK-18527
> URL: https://issues.apache.org/jira/browse/SPARK-18527
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0, 2.0.1
> Environment: spark-2.0.1-bin-hadoop2.7/bin/spark-shell
> Reporter: Fabian Boehnlein
> Assignee: Herman van Hovell
> Fix For: 2.1.0
>
>
> Same bug as SPARK-16228 but
> {code}_FUNC_(bigint, array<double>) {code}
> instead of
> {code}_FUNC_(bigint, double){code}
> Fix of SPARK-16228 only fixes the non-array case that was hit.
> {code}
> sql("select percentile(value, array(0.5,0.99)) from values 1,2,3 T(value)")
> {code}
> fails in Spark 2 shell.
> Longer example
> {code}
> case class Record(key: Long, value: String)
> val recordsDF = spark.createDataFrame((1 to 100).map(i => Record(i.toLong, s"val_$i")))
> recordsDF.createOrReplaceTempView("records")
> sql("SELECT percentile(key, Array(0.95, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1)) AS test FROM records")
> org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'org.apache.hadoop.hive.ql.udf.UDAFPercentile': org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method for class org.apache.had
> oop.hive.ql.udf.UDAFPercentile with (bigint, array<decimal(38,18)>). Possible choices: _FUNC_(bigint, array<double>) _FUNC_(bigint, double) ; line 1 pos 7
> at org.apache.hadoop.hive.ql.exec.FunctionRegistry.getMethodInternal(FunctionRegistry.java:1164)
> at org.apache.hadoop.hive.ql.exec.DefaultUDAFEvaluatorResolver.getEvaluatorClass(DefaultUDAFEvaluatorResolver.java:83)
> at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge.getEvaluator(GenericUDAFBridge.java:56)
> at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org