You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jianshi Huang <ji...@gmail.com> on 2015/01/13 22:51:36 UTC
Hive UDAF percentile_approx says "This UDAF does not support the
deprecated getEvaluator() method."
Hi,
The following SQL query
select percentile_approx(variables.var1, 0.95) p95
from model
will throw
ERROR SparkSqlInterpreter: Error
org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
support the deprecated getEvaluator() method.
at
org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
at
org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
at
org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
at
org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
at
org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
at
org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
I'm using latest branch-1.2
I found in PR that percentile and percentile_approx are supported. A bug?
Thanks,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Hive UDAF percentile_approx says "This UDAF does not support the
deprecated getEvaluator() method."
Posted by Jianshi Huang <ji...@gmail.com>.
Ah, thx Ted and Yin!
I'll build a new version. :)
Jianshi
On Wed, Jan 14, 2015 at 7:24 AM, Yin Huai <yh...@databricks.com> wrote:
> Yeah, it's a bug. It has been fixed by
> https://issues.apache.org/jira/browse/SPARK-3891 in master.
>
> On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Looking at the source code for AbstractGenericUDAFResolver, the following
>> (non-deprecated) method should be called:
>>
>> public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info)
>>
>> It is called by hiveUdfs.scala (master branch):
>>
>> val parameterInfo = new
>> SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false)
>> resolver.getEvaluator(parameterInfo)
>>
>> FYI
>>
>> On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang <ji...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The following SQL query
>>>
>>> select percentile_approx(variables.var1, 0.95) p95
>>> from model
>>>
>>> will throw
>>>
>>> ERROR SparkSqlInterpreter: Error
>>> org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
>>> support the deprecated getEvaluator() method.
>>> at
>>> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
>>> at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
>>> at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
>>> at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
>>> at
>>> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
>>> at
>>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>> at
>>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>>
>>> I'm using latest branch-1.2
>>>
>>> I found in PR that percentile and percentile_approx are supported. A bug?
>>>
>>> Thanks,
>>> --
>>> Jianshi Huang
>>>
>>> LinkedIn: jianshi
>>> Twitter: @jshuang
>>> Github & Blog: http://huangjs.github.com/
>>>
>>
>>
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Hive UDAF percentile_approx says "This UDAF does not support the
deprecated getEvaluator() method."
Posted by Yin Huai <yh...@databricks.com>.
Yeah, it's a bug. It has been fixed by
https://issues.apache.org/jira/browse/SPARK-3891 in master.
On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu <yu...@gmail.com> wrote:
> Looking at the source code for AbstractGenericUDAFResolver, the following
> (non-deprecated) method should be called:
>
> public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info)
>
> It is called by hiveUdfs.scala (master branch):
>
> val parameterInfo = new
> SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false)
> resolver.getEvaluator(parameterInfo)
>
> FYI
>
> On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> Hi,
>>
>> The following SQL query
>>
>> select percentile_approx(variables.var1, 0.95) p95
>> from model
>>
>> will throw
>>
>> ERROR SparkSqlInterpreter: Error
>> org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
>> support the deprecated getEvaluator() method.
>> at
>> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
>> at
>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
>> at
>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
>> at
>> org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
>> at
>> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
>> at
>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>> at
>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>
>> I'm using latest branch-1.2
>>
>> I found in PR that percentile and percentile_approx are supported. A bug?
>>
>> Thanks,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
Re: Hive UDAF percentile_approx says "This UDAF does not support the
deprecated getEvaluator() method."
Posted by Ted Yu <yu...@gmail.com>.
Looking at the source code for AbstractGenericUDAFResolver, the following
(non-deprecated) method should be called:
public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info)
It is called by hiveUdfs.scala (master branch):
val parameterInfo = new
SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false)
resolver.getEvaluator(parameterInfo)
FYI
On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> Hi,
>
> The following SQL query
>
> select percentile_approx(variables.var1, 0.95) p95
> from model
>
> will throw
>
> ERROR SparkSqlInterpreter: Error
> org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
> support the deprecated getEvaluator() method.
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
> at
> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
> at
> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
> at
> org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
> at
> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
> at
> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
> at
> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>
> I'm using latest branch-1.2
>
> I found in PR that percentile and percentile_approx are supported. A bug?
>
> Thanks,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>