You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mark Andreev <ma...@gmail.com> on 2026/08/10 20:36:22 UTC
[DISCUSS] Support GROUPS window frames in Spark SQL
Hi all,
I’d like to propose adding SQL-standard `GROUPS` window frames to Spark SQL.
Spark currently supports `ROWS` and `RANGE`, but not `GROUPS`, which
defines frame boundaries in terms of peer groups. This is useful where
group sizes and ordering-key gaps vary, and cannot in general be expressed
correctly with a fixed `ROWS` or `RANGE` frame.
I’ve prepared a design proposal covering semantics, compatibility,
implementation feasibility, testing, and behaviour across other SQL engines:
https://docs.google.com/document/d/1BOVWBHz6vQi39uQ_8SJfMWg55oJrUqNMQAgtHXmTZhA/edit?usp=sharing
I intend to implement this and open a draft PR.
Before starting, I’d particularly appreciate feedback on:
* whether this should go through SPIP or a normal JIRA + PR;
* whether `GROUPS` support for Python/Pandas window UDAFs should be
included initially or deferred.
Any feedback on the proposed semantics or implementation direction would be
very welcome.
--
Best regards,
Mark Andreev
Re: [DISCUSS] Support GROUPS window frames in Spark SQL
Posted by Mark Andreev <ma...@gmail.com>.
Hi Uroš,
Could you review https://github.com/apache/spark/pull/58634
( [SPARK-58980][SQL] Support GROUPS window frames ) when you have a chance?
I’d be happy to address any feedback or make any changes you suggest.
On Sun, 23 Aug 2026 at 18:40, Uroš Bojanić <ur...@apache.org> wrote:
> P.S. Please check existing open issues on the Spark Jira, e.g. I found
> https://issues.apache.org/jira/browse/SPARK-28648, which seems to
> describe the same feature.
>
> However, it's been dormant for quite a while now, and the original comment
> objection rationale there is outdated, since other systems now support
> GROUPS too.
>
> Also, this item is a sub-task of SPARK-27764 "Feature Parity between
> PostgreSQL and Spark", so you're probably fine creating a new Jira ID for
> your proposal.
>
> On 2026/08/10 20:36:22 Mark Andreev wrote:
> > Hi all,
> >
> > I’d like to propose adding SQL-standard `GROUPS` window frames to Spark
> SQL.
> >
> > Spark currently supports `ROWS` and `RANGE`, but not `GROUPS`, which
> > defines frame boundaries in terms of peer groups. This is useful where
> > group sizes and ordering-key gaps vary, and cannot in general be
> expressed
> > correctly with a fixed `ROWS` or `RANGE` frame.
> >
> > I’ve prepared a design proposal covering semantics, compatibility,
> > implementation feasibility, testing, and behaviour across other SQL
> engines:
> >
> >
> https://docs.google.com/document/d/1BOVWBHz6vQi39uQ_8SJfMWg55oJrUqNMQAgtHXmTZhA/edit?usp=sharing
> >
> > I intend to implement this and open a draft PR.
> >
> > Before starting, I’d particularly appreciate feedback on:
> >
> > * whether this should go through SPIP or a normal JIRA + PR;
> > * whether `GROUPS` support for Python/Pandas window UDAFs should be
> > included initially or deferred.
> >
> > Any feedback on the proposed semantics or implementation direction would
> be
> > very welcome.
> >
> > --
> > Best regards,
> > Mark Andreev
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Best regards,
Mark Andreev
Re: [DISCUSS] Support GROUPS window frames in Spark SQL
Posted by Uroš Bojanić <ur...@apache.org>.
P.S. Please check existing open issues on the Spark Jira, e.g. I found https://issues.apache.org/jira/browse/SPARK-28648, which seems to describe the same feature.
However, it's been dormant for quite a while now, and the original comment objection rationale there is outdated, since other systems now support GROUPS too.
Also, this item is a sub-task of SPARK-27764 "Feature Parity between PostgreSQL and Spark", so you're probably fine creating a new Jira ID for your proposal.
On 2026/08/10 20:36:22 Mark Andreev wrote:
> Hi all,
>
> I’d like to propose adding SQL-standard `GROUPS` window frames to Spark SQL.
>
> Spark currently supports `ROWS` and `RANGE`, but not `GROUPS`, which
> defines frame boundaries in terms of peer groups. This is useful where
> group sizes and ordering-key gaps vary, and cannot in general be expressed
> correctly with a fixed `ROWS` or `RANGE` frame.
>
> I’ve prepared a design proposal covering semantics, compatibility,
> implementation feasibility, testing, and behaviour across other SQL engines:
>
> https://docs.google.com/document/d/1BOVWBHz6vQi39uQ_8SJfMWg55oJrUqNMQAgtHXmTZhA/edit?usp=sharing
>
> I intend to implement this and open a draft PR.
>
> Before starting, I’d particularly appreciate feedback on:
>
> * whether this should go through SPIP or a normal JIRA + PR;
> * whether `GROUPS` support for Python/Pandas window UDAFs should be
> included initially or deferred.
>
> Any feedback on the proposed semantics or implementation direction would be
> very welcome.
>
> --
> Best regards,
> Mark Andreev
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] Support GROUPS window frames in Spark SQL
Posted by Uroš Bojanić <ur...@apache.org>.
Hi Mark,
The proposal sounds good to me, +1 as it's a real gap in Spark. GROUPS is SQL:2011.
Also, this should be feasible, given that GROUPS is a relatively close cousin of RANGE. It will be a well scoped, standard defined, additive change. No broad arch changes.
Your design doc looks good, please attach it to the Jira, for any further discussions. Regarding scope, Python can be deferred initially, and done in an immediate followup.
With this in mind, I think a normal Jira and PR will be fine here. No need for SPIP.
Thank you for the well scoped proposal, looking forward to the seeing the PR!
Best,
Uroš
On 2026/08/10 20:36:22 Mark Andreev wrote:
> Hi all,
>
> I’d like to propose adding SQL-standard `GROUPS` window frames to Spark SQL.
>
> Spark currently supports `ROWS` and `RANGE`, but not `GROUPS`, which
> defines frame boundaries in terms of peer groups. This is useful where
> group sizes and ordering-key gaps vary, and cannot in general be expressed
> correctly with a fixed `ROWS` or `RANGE` frame.
>
> I’ve prepared a design proposal covering semantics, compatibility,
> implementation feasibility, testing, and behaviour across other SQL engines:
>
> https://docs.google.com/document/d/1BOVWBHz6vQi39uQ_8SJfMWg55oJrUqNMQAgtHXmTZhA/edit?usp=sharing
>
> I intend to implement this and open a draft PR.
>
> Before starting, I’d particularly appreciate feedback on:
>
> * whether this should go through SPIP or a normal JIRA + PR;
> * whether `GROUPS` support for Python/Pandas window UDAFs should be
> included initially or deferred.
>
> Any feedback on the proposed semantics or implementation direction would be
> very welcome.
>
> --
> Best regards,
> Mark Andreev
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org