You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Wiśniowski Piotr <co...@gmail.com> on 2024/03/06 14:17:34 UTC
Updating streaming DataFlow pipeline
Hi
I have a question. I do have a long running streaming dataflow jobs done
with python sdk. I did notice that when I do update a job with exactly
same code that created it, it may randomly fail (~50/50) with error like
`The Coder or type for step event_stats/join/dewindow.None/FromValue has
changed`.
The `dewindow` step is just `WindowInto(GlobalWinsows()` so that
downstream state-full `DoFn` would have its state global per key. I am
pretty sure that I did not change any types (as this is exact same code)
and the coders I use are deterministic (pydantic avro) and types are set
for every `PCollection`. What else should I look into, which could
result in such behavior?
Best
Wiśniowski Piotr