[jira] [Commented] (FLINK-7799) Improve performance of windowed joins

["Xingcan Cui (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204554#comment-16204554 ] 

Xingcan Cui commented on FLINK-7799:
------------------------------------

Hi [~fhueske], I've reconsidered this problem and found a drawback of the time block solution. 

Specifically, if we merge all the rows belonging to the same time block in an entry (the value of which is either a {{Map}} or a {{List}}) of the {{MapState}}, the minimum operating unit of the state becomes a collection. That means everytime we store/remove a single row, all the data in the same block must also be rewritten, which will definitely bring a lot of extra cost.

If that drawback cannot be eliminated, I wonder if we could improve the join performance from another point of view. Since the rocksdb backend should be widely used in real applications and the {{MapState}} entries are ordered in it, can we provide something like a  hint mechanism in the state API, so that the join function can be aware of the ordering?

> Improve performance of windowed joins
> -------------------------------------
>
>                 Key: FLINK-7799
>                 URL: https://issues.apache.org/jira/browse/FLINK-7799
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table API & SQL
>    Affects Versions: 1.4.0
>            Reporter: Fabian Hueske
>            Assignee: Xingcan Cui
>            Priority: Critical
>
> The performance of windowed joins can be improved by changing the state access patterns.
> Right now, rows are inserted into a MapState with their timestamp as key. Since we use a time resolution of 1ms, this means that the full key space of the state must be iterated and many map entries must be accessed when joining or evicting rows. 
> A better strategy would be to block the time into larger intervals and register the rows in their respective interval. Another benefit would be that we can directly access the state entries because we know exactly which timestamps to look up. Hence, we can limit the state access to the relevant section during joining and state eviction. 
> The good size for intervals needs to be identified and might depend on the size of the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)