[D] Advices for implementing Apache Flink like windows based on DataFusion [datafusion]

["niebayes (via GitHub)" <gi...@apache.org>]

GitHub user niebayes created a discussion: Advices for implementing Apache Flink like windows based on DataFusion

Streaming computation engines, such as Apache Flink, support various types of windows. Each window type has a window assigner, which determines how to partition time-series data into windows, and then performs computations within those windows. Commonly used window assigners include tumbling windows, sliding windows, session windows, count windows, and state windows.

I would like to implement these window types based on DataFusion. Does DataFusion’s existing infrastructure—such as UDWFs (User-Defined Window Functions)—support this use case? Do you have any suggestions for how to implement this?

Brief overview of window types:

- Tumbling Window: Fixed-size, non-overlapping time intervals (e.g., every 5 minutes).
- Sliding Window: Fixed-size windows that can overlap, sliding by a smaller step (e.g., 5-minute windows sliding every 1 minute).
- Session Window: Dynamic windows based on periods of activity separated by inactivity (gaps).
- Count Window: Windows based on a fixed number of events instead of time (e.g., every 100 records).
- State Window: Custom windows that are determined by data state or logic rather than time or count.

GitHub link: https://github.com/apache/datafusion/discussions/16400

----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org