[jira] [Resolved] (SPARK-22480) Dynamic Watermarking

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-22480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-22480.
----------------------------------
    Resolution: Incomplete

> Dynamic Watermarking
> --------------------
>
>                 Key: SPARK-22480
>                 URL: https://issues.apache.org/jira/browse/SPARK-22480
>             Project: Spark
>          Issue Type: Wish
>          Components: Structured Streaming
>    Affects Versions: 2.2.0
>            Reporter: Jochen Niebuhr
>            Priority: Major
>              Labels: bulk-closed
>
> When you're using the watermark feature, you're forced to provide an absolute duration to identify late events. For the case we're using structured streaming for, this is not completely working. In our case, late events will be possible on the next business day. So I'd like to use 24 hours watermark for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would suggest is being able to use a function or expression to {{withWatermark}} so people can implement this or similar behaviors. If this sounds like a good idea, I can probably supply a pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org