[jira] [Commented] (BEAM-12131) DataFrame API: Integrate pandas windowing with Beam event-time semantics

["Brian Hulette (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353580#comment-17353580 ] 

Brian Hulette commented on BEAM-12131:
--------------------------------------

This is blocked on adding native support for columnar/vectorized DoFns in Beam. We need to be able to specify a timestamp/window for elements within the DataFrame independently, which is incompatible with the current model where an entire DataFrame with multiple elements is a single logical Beam element, with one timestamp and window.

> DataFrame API: Integrate pandas windowing with Beam event-time semantics
> ------------------------------------------------------------------------
>
>                 Key: BEAM-12131
>                 URL: https://issues.apache.org/jira/browse/BEAM-12131
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: Brian Hulette
>            Priority: P2
>              Labels: dataframe-api
>
> pandas has support for [rolling()|https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html] windows. It should be possible to use this within a DataframeTransform and have it apply Beam windowing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)