You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Wiśniowski Piotr <co...@gmail.com> on 2024/03/06 14:17:34 UTC

Updating streaming DataFlow pipeline

Hi

I have a question. I do have a long running streaming dataflow jobs done 
with python sdk. I did notice that when I do update a job with exactly 
same code that created it, it may randomly fail (~50/50) with error like 
`The Coder or type for step event_stats/join/dewindow.None/FromValue has 
changed`.

The `dewindow` step is just `WindowInto(GlobalWinsows()` so that 
downstream state-full `DoFn`  would have its state global per key.  I am 
pretty sure that I did not change any types (as this is exact same code) 
and the coders I use are deterministic (pydantic avro) and types are set 
for every `PCollection`. What else should I look into, which could 
result in such behavior?

Best

Wiśniowski Piotr