You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Something Something <ma...@gmail.com> on 2020/05/07 18:42:56 UTC

Dynamically changing maxOffsetsPerTrigger

Is there a way to dynamically modify value of 'maxOffsetsPerTrigger' while
a Stateful Structured Streaming job is running?

We are thinking of auto-scaling our Spark cluster but if we don't modify
the value of 'maxOffsetsPerTrigger' dynamically would adding more VMs to
the cluster help? I don't think it would, would it?

In other words, if I add 2 new VMs to the cluster but value of
'maxOffsetsPerTrigger' is still the same would performance improve? I would
think not. We would have to explicitly stop the job, add VMs & then restart
the job after changing the value of 'maxOffsetsPerTrigger' - which defeats
the purpose of Auto-scaling.

Please tell me if my understanding is not correct. Thanks.