How to execute non-timestamp-based aggregations in spark structured streaming?

[Stephen Boesch <ja...@gmail.com>]

Consider the following *intended* sql:

select row_number()
  over (partition by Origin order by OnTimeDepPct desc) OnTimeDepRank,*
  from flights

This will *not* work in *structured streaming* : The culprit is:

 partition by Origin

The requirement is to use a timestamp-typed field such as

 partition by flightTime

Tathagata Das (core committer for *spark streaming*) - replies on that in a
nabble thread:

 The traditional SQL windows with `over` is not supported in streaming.
Only time-based windows, that is, `window("timestamp", "10 minutes")` is
supported in streaming

*W**hat then* for my query above - which *must* be based on the *Origin* field?
What is the closest equivalent to that query? Or what would be a workaround
or different approach to achieve same results?

Re: How to execute non-timestamp-based aggregations in spark structured streaming?

[Tathagata Das <td...@databricks.com>]

SQL windows with the 'over' syntax does not work in Structured Streaming.
It is very hard to incrementalize that in the general case. Hence non-time
windows are not supported.

On Sat, Apr 20, 2019, 2:16 PM Stephen Boesch <ja...@gmail.com> wrote:

> Consider the following *intended* sql:
>
> select row_number()
>   over (partition by Origin order by OnTimeDepPct desc) OnTimeDepRank,*
>   from flights
>
> This will *not* work in *structured streaming* : The culprit is:
>
>  partition by Origin
>
> The requirement is to use a timestamp-typed field such as
>
>  partition by flightTime
>
> Tathagata Das (core committer for *spark streaming*) - replies on that in
> a nabble thread:
>
>  The traditional SQL windows with `over` is not supported in streaming.
> Only time-based windows, that is, `window("timestamp", "10 minutes")` is
> supported in streaming
>
> *W**hat then* for my query above - which *must* be based on the *Origin* field?
> What is the closest equivalent to that query? Or what would be a workaround
> or different approach to achieve same results?
>