You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2016/03/06 04:00:45 UTC
[jira] [Comment Edited] (SPARK-12718) SQL generation support for
window functions
[ https://issues.apache.org/jira/browse/SPARK-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181967#comment-15181967 ]
Xiao Li edited comment on SPARK-12718 at 3/6/16 2:59 AM:
---------------------------------------------------------
{code}
select key, value, avg(c_int) over (partition by key), sum(c_float) over(partition by value) from t1
{code}
->
{code}
select key, value, avg(c_int), t2._w0 over (partition by key) from (select key, value, sum(c_float) over(partition by value) as _w0 from t1 where value < 10) t2
{code}
When window specifications are different, we will split the whole one to multiple. However, to do it, we need to have the corresponding optimizer rule to combine them back. Let me check if we need to add a rule.
In addition, I plan to add the predicate pushdown for window into Optimizer first. This is still missing now. I also found Hive just added one. Thanks!
was (Author: smilegator):
select key, value, avg(c_int) over (partition by key), sum(c_float) over(partition by value) from t1
->
select key, value, avg(c_int), t2._w0 over (partition by key) from (select key, value, sum(c_float) over(partition by value) as _w0 from t1 where value < 10) t2
When window specifications are different, we will split the whole one to multiple. However, to do it, we need to have the corresponding optimizer rule to combine them back. Let me check if we need to add a rule.
In addition, I plan to add the predicate pushdown for window into Optimizer first. This is still missing now. I also found Hive just added one. Thanks!
> SQL generation support for window functions
> -------------------------------------------
>
> Key: SPARK-12718
> URL: https://issues.apache.org/jira/browse/SPARK-12718
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Cheng Lian
> Assignee: Xiao Li
>
> {{HiveWindowFunctionQuerySuite}} and {{HiveWindowFunctionQueryFileSuite}} can be useful for bootstrapping test coverage. Please refer to SPARK-11012 for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org