You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/07/06 01:43:18 UTC
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14136
Hi @rxin, I was reading related codes around this and saw - https://github.com/apache/spark/pull/14136#discussion_r84591480.
It looks many suggestions for calculating median are workarounds (e.g., https://stackoverflow.com/a/31437177).
I want to use `approximate_percentile` or `percentile` in groupby/pivot and I tried to deal with this problem for few days. I ended up with a weird code as below, e.g.,:
```python
from pyspark.sql.functions import *
from pyspark.sql.column import Column, _to_java_column
def approximate_percentile(child, percentage, accuracy=lit(10000)):
percentile_expr = spark.sparkContext._jvm.org.apache.spark.sql.catalyst.expressions.aggregate.ApproximatePercentile
child_expr = _to_java_column(child).expr()
percentage_expr = _to_java_column(percentage).expr()
accuracy_expr = _to_java_column(accuracy).expr()
agg_func = percentile_expr(child_expr, percentage_expr, accuracy_expr)
return Column(spark._jvm.org.apache.spark.sql.Column(agg_func.toAggregateExpression()))
df = spark.range(1)
df.select(approximate_percentile(df.id, lit(0.5))).show()
```
This code might be easily broken by Spark version as it accesses to internal packages via JVM. I use
Scala/R but also PySpark specifically to avoid version compatibility problem (even between Spark 1.6.x and Spark 2.x) in many cases but this one now becomes flaky.
Another alternative should be to port existing logic in application side to SQL ones but I was wondering if I really should do this for single case.
It might be expensive but exposing it might also promote users to test this at least.
Could we expose this in Scala/Python/R? It should be pretty easy to expose this. Or, did I misunderstand the context and other workarounds?
cc @srowen and @zero323 who I saw answered to the questions related with this outside (e.g., stackoverflow).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org