Re: Autoscaling stuck at 1, never see getSplitBacklogBytes() execute

[Ken Barr <Ke...@Solace.com>]

Does anyone actually use Streaming Autoscaling with cloud Dataflow?  I have seen scale-ups based on CPU but never on backlog.  Now I do not see scale up events at all.  If this works can you please point me to a working example.

On 2019/01/09 20:09:46, Ken Barr <Ke...@solace.com> wrote: 
> Hello
> 
> I have been directed to this mail list by posting to Stack Overflow.  If this is inappropriate use of mail list please let me know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
> 
> I am creating a custom IO connector in Java that runs is streaming mode with 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, similar to JMS IO. All works well except the auto-scaling does not seem to work. The odd part for me is that I have implemented a getSplitBacklogBytes() method, but as far as the logs show, it is never executed.
> 
> I am testing with a modified wordcount example to test, executed with:
> 
> --runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED  --maxNumWorkers=4
> 
> 
> 
> The Pipeline options and logs seem to show that auto-scaling is being set up correctly.
> 
> PipelineOptions: Current Settings:
> 
>   autoscalingAlgorithm: THROUGHPUT_BASED
> 
>   maxNumWorkers: 4
> 
>   numWorkers: 0
> 
> Logs show:
> 
> 20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will be between 1 and 4.
> 
> 20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker processes.
> 
> 20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent disks which will allow autoscaling from 1 to 4 workers.
> 
> 20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers...
> 
> 
> 
> Is there anything else that needs to be done to enable this feature, or anything I might have done to inadvertently disable this feature?
> 
> Source code lives here: https://github.com/SolaceLabs/solace-beam-integration/tree/development
> 
>