You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Something Something <ma...@gmail.com> on 2020/05/07 18:42:56 UTC
Dynamically changing maxOffsetsPerTrigger
Is there a way to dynamically modify value of 'maxOffsetsPerTrigger' while
a Stateful Structured Streaming job is running?
We are thinking of auto-scaling our Spark cluster but if we don't modify
the value of 'maxOffsetsPerTrigger' dynamically would adding more VMs to
the cluster help? I don't think it would, would it?
In other words, if I add 2 new VMs to the cluster but value of
'maxOffsetsPerTrigger' is still the same would performance improve? I would
think not. We would have to explicitly stop the job, add VMs & then restart
the job after changing the value of 'maxOffsetsPerTrigger' - which defeats
the purpose of Auto-scaling.
Please tell me if my understanding is not correct. Thanks.