You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/01/15 02:15:27 UTC
[jira] [Commented] (SPARK-17458) Alias specified for aggregates in
a pivot are not honored
[ https://issues.apache.org/jira/browse/SPARK-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823010#comment-15823010 ]
Apache Spark commented on SPARK-17458:
--------------------------------------
User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/16565
> Alias specified for aggregates in a pivot are not honored
> ---------------------------------------------------------
>
> Key: SPARK-17458
> URL: https://issues.apache.org/jira/browse/SPARK-17458
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Ravi Somepalli
> Assignee: Andrew Ray
> Fix For: 2.1.0
>
>
> When using pivot and multiple aggregations we need to alias to avoid special characters, but alias does not help because
> df.groupBy("C").pivot("A").agg(avg("D").as("COLD"), max("B").as("COLB")).show
> || C || bar_avg(`D`) AS `COLD` || bar_max(`B`) AS `COLB` || foo_avg(`D`) AS `COLD` || foo_max(`B`) AS `COLB` ||
> |small| 5.5| two| 2.3333333333333335| two|
> |large| 5.5| two| 2.0| one|
> Expected Output
> || C || bar_COLD || bar_COLB || foo_COLD || foo_COLB ||
> |small| 5.5| two| 2.3333333333333335| two|
> |large| 5.5| two| 2.0| one|
> One approach you can fix this issue is to change the class
> sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
> and change the outputName method in
> {code}
> object ResolvePivot extends Rule[LogicalPlan] {
> def apply(plan: LogicalPlan): LogicalPlan = plan transform {
> {code}
> {code}
> def outputName(value: Literal, aggregate: Expression): String = {
> val suffix = aggregate match {
> case n: NamedExpression => aggregate.asInstanceOf[NamedExpression].name
> case _ => aggregate.sql
> }
> if (singleAgg) value.toString else value + "_" + suffix
> }
> {code}
> Version : 2.0.0
> {code}
> def outputName(value: Literal, aggregate: Expression): String = {
> if (singleAgg) value.toString else value + "_" + aggregate.sql
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org