You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Roman Khachatryan (Jira)" <ji...@apache.org> on 2022/02/09 00:35:01 UTC

[jira] [Created] (FLINK-26019) Changelogged PriorityQueue elements recovered out-of-order

Roman Khachatryan created FLINK-26019:
-----------------------------------------

             Summary: Changelogged PriorityQueue elements recovered out-of-order
                 Key: FLINK-26019
                 URL: https://issues.apache.org/jira/browse/FLINK-26019
             Project: Flink
          Issue Type: Bug
          Components: Runtime / State Backends
    Affects Versions: 1.15.0
            Reporter: Roman Khachatryan
            Assignee: Roman Khachatryan
             Fix For: 1.15.0


StateChangeFormat is the class responsible for writing out changelog data.
Each chunk of data is sorted by: logId -> sequenceNumber -> keyGroup.
Sorting by sequenceNumber preserves temporal order.
Sorting by keyGroup a) puts metadata (group -1) at the beginning and b) allows to write KG only once.

However, the assumption that the order of changes across groups currently doesn't hold: poll operation of InternalPriorityQueue may affect any group (the smaller item across groups so far will be polled).

This results in wrong processing time timers being removed on recovery in ProcessingTimeWindowCheckpointingITCase#testAggregatingSlidingProcessingTimeWindow

One way to solve this probelm is to simply disable KG-sorting and grouping (only output metadata at the beginning). 
The other one is to associate polled element with the correct key group while logging changes.

Both ways should work with re-scaling.

cc: [~masteryhx], [~ym], [~yunta]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)