You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Raghu Vemuri via user <us...@beam.apache.org> on 2026/08/17 06:29:00 UTC

[Bug]: Java FlinkRunner OrderedListState silently overwrites duplicate elements sharing same timestamps

## Componentrunner-flink
## What happened?When executing a pipeline using the `FlinkRunner` (specifically observed using the `HashMapStateBackend` / Java heap state configuration), adding multiple distinct elements with the **exact same timestamp** to an `OrderedListState` results in critical data loss. The last element written completely overwrites any previous elements that shared that specific millisecond timestamp bucket.
According to Beam's core architectural contract, `OrderedListState` should preserve and sort all appended values. It does not impose an invariant rule requiring timestamps to be globally unique. 
The underlying flaw stems from how `FlinkStateInternals` maps `OrderedListState` onto Flink's native primitive structures. The runner sets the key of Flink's internal tracking storage structure to the `Long` timestamp value itself. During back-to-back writes in a single execution bundle, the internal write-buffer performs a flat replacement (equivalent to a map `put` action) instead of appending the element to a sub-collection (like a `List`) associated with that timestamp bucket.
## Expected Behavior`OrderedListState` should treat the timestamp solely as a sorting key index. If multiple elements share a timestamp, the runner's underlying serialization layout must wrap elements in an appendable multi-value collection (e.g., mapping to a Flink `MapState<Long, List<T>>`) so that subsequent writes do not execute a flat destructive overwrite.
## Actual BehaviorThe second write (`"Ball"`) silently destroys and overwrites the initial write (`"Apple"`) at the timestamp boundary level inside the Java heap state backend.
## Environment**Apache Beam Version:** 2.61.0**Flink Version:** 1.19.1**Runner:** FlinkRunner**State Backend:** `HashMapStateBackend` (Java Heap)

Re: [Bug]: Java FlinkRunner OrderedListState silently overwrites duplicate elements sharing same timestamps

Posted by Raghu Vemuri via user <us...@beam.apache.org>.
 Here is the bug - https://github.com/apache/beam/issues/39782
    On Monday, August 17, 2026 at 01:02:31 AM PDT, Jan Lukavský <je...@seznam.cz> wrote:  
 
  
Hi,
 
confirming this looks like a bug. FlinkStateInternals.FlinkOrderedListState.readAsMap keys elements by Instant, which is not guaranteed to be unique. Can you please fill a github issue for this?
 
 Jan
 On 8/17/26 08:29, Raghu Vemuri via user wrote:
  
    ## Component runner-flink  
    ## What happened? When executing a pipeline using the `FlinkRunner` (specifically observed using the `HashMapStateBackend` / Java heap state configuration), adding multiple distinct elements with the **exact same timestamp** to an `OrderedListState` results in critical data loss. The last element written completely overwrites any previous elements that shared that specific millisecond timestamp bucket.  
    According to Beam's core architectural contract, `OrderedListState` should preserve and sort all appended values. It does not impose an invariant rule requiring timestamps to be globally unique.  
  The underlying flaw stems from how `FlinkStateInternals` maps `OrderedListState` onto Flink's native primitive structures. The runner sets the key of Flink's internal tracking storage structure to the `Long` timestamp value itself. During back-to-back writes in a single execution bundle, the internal write-buffer performs a flat replacement (equivalent to a map `put` action) instead of appending the element to a sub-collection (like a `List`) associated with that timestamp bucket.  
    ## Expected Behavior `OrderedListState` should treat the timestamp solely as a sorting key index. If multiple elements share a timestamp, the runner's underlying serialization layout must wrap elements in an appendable multi-value collection (e.g., mapping to a Flink `MapState<Long, List<T>>`) so that subsequent writes do not execute a flat destructive overwrite. 
  ## Actual Behavior The second write (`"Ball"`) silently destroys and overwrites the initial write (`"Apple"`) at the timestamp boundary level inside the Java heap state backend. 
    ## Environment **Apache Beam Version:** 2.61.0 **Flink Version:** 1.19.1 **Runner:** FlinkRunner **State Backend:** `HashMapStateBackend` (Java Heap)   
        
   

Re: [Bug]: Java FlinkRunner OrderedListState silently overwrites duplicate elements sharing same timestamps

Posted by Jan Lukavský <je...@seznam.cz>.
Hi,

confirming this looks like a bug. 
FlinkStateInternals.FlinkOrderedListState.readAsMap keys elements by 
Instant, which is not guaranteed to be unique. Can you please fill a 
github issue for this?

  Jan

On 8/17/26 08:29, Raghu Vemuri via user wrote:
> ## Component
> runner-flink
>
> ## What happened?
> When executing a pipeline using the `FlinkRunner` (specifically 
> observed using the `HashMapStateBackend` / Java heap state 
> configuration), adding multiple distinct elements with the **exact 
> same timestamp** to an `OrderedListState` results in critical data 
> loss. The last element written completely overwrites any previous 
> elements that shared that specific millisecond timestamp bucket.
>
> According to Beam's core architectural contract, `OrderedListState` 
> should preserve and sort all appended values. It does not impose an 
> invariant rule requiring timestamps to be globally unique.
>
> The underlying flaw stems from how `FlinkStateInternals` maps 
> `OrderedListState` onto Flink's native primitive structures. The 
> runner sets the key of Flink's internal tracking storage structure to 
> the `Long` timestamp value itself. During back-to-back writes in a 
> single execution bundle, the internal write-buffer performs a flat 
> replacement (equivalent to a map `put` action) instead of appending 
> the element to a sub-collection (like a `List`) associated with that 
> timestamp bucket.
>
> ## Expected Behavior
> `OrderedListState` should treat the timestamp solely as a sorting key 
> index. If multiple elements share a timestamp, the runner's underlying 
> serialization layout must wrap elements in an appendable multi-value 
> collection (e.g., mapping to a Flink `MapState<Long, List<T>>`) so 
> that subsequent writes do not execute a flat destructive overwrite.
>
> ## Actual Behavior
> The second write (`"Ball"`) silently destroys and overwrites the 
> initial write (`"Apple"`) at the timestamp boundary level inside the 
> Java heap state backend.
>
> ## Environment
> **Apache Beam Version:** 2.61.0
> **Flink Version:** 1.19.1
> **Runner:** FlinkRunner
> **State Backend:** `HashMapStateBackend` (Java Heap)
>