You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ken Barr <Ke...@solace.com> on 2019/01/09 20:09:46 UTC
Autoscaling stuck at 1, never see getSplitBacklogBytes() execute
Hello
I have been directed to this mail list by posting to Stack Overflow. If this is inappropriate use of mail list please let me know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
I am creating a custom IO connector in Java that runs is streaming mode with 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, similar to JMS IO. All works well except the auto-scaling does not seem to work. The odd part for me is that I have implemented a getSplitBacklogBytes() method, but as far as the logs show, it is never executed.
I am testing with a modified wordcount example to test, executed with:
--runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=4
The Pipeline options and logs seem to show that auto-scaling is being set up correctly.
PipelineOptions: Current Settings:
autoscalingAlgorithm: THROUGHPUT_BASED
maxNumWorkers: 4
numWorkers: 0
Logs show:
20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will be between 1 and 4.
20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker processes.
20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent disks which will allow autoscaling from 1 to 4 workers.
20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers...
Is there anything else that needs to be done to enable this feature, or anything I might have done to inadvertently disable this feature?
Source code lives here: https://github.com/SolaceLabs/solace-beam-integration/tree/development
Re: Autoscaling stuck at 1, never see getSplitBacklogBytes() execute
Posted by Ken Barr <Ke...@Solace.com>.
Does anyone actually use Streaming Autoscaling with cloud Dataflow? I have seen scale-ups based on CPU but never on backlog. Now I do not see scale up events at all. If this works can you please point me to a working example.
On 2019/01/09 20:09:46, Ken Barr <Ke...@solace.com> wrote:
> Hello
>
> I have been directed to this mail list by posting to Stack Overflow. If this is inappropriate use of mail list please let me know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
>
> I am creating a custom IO connector in Java that runs is streaming mode with 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, similar to JMS IO. All works well except the auto-scaling does not seem to work. The odd part for me is that I have implemented a getSplitBacklogBytes() method, but as far as the logs show, it is never executed.
>
> I am testing with a modified wordcount example to test, executed with:
>
> --runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=4
>
>
>
> The Pipeline options and logs seem to show that auto-scaling is being set up correctly.
>
> PipelineOptions: Current Settings:
>
> autoscalingAlgorithm: THROUGHPUT_BASED
>
> maxNumWorkers: 4
>
> numWorkers: 0
>
> Logs show:
>
> 20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will be between 1 and 4.
>
> 20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker processes.
>
> 20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent disks which will allow autoscaling from 1 to 4 workers.
>
> 20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers...
>
>
>
> Is there anything else that needs to be done to enable this feature, or anything I might have done to inadvertently disable this feature?
>
> Source code lives here: https://github.com/SolaceLabs/solace-beam-integration/tree/development
>
>
Re: Autoscaling stuck at 1, never see getSplitBacklogBytes() execute
Posted by Ken Barr <Ke...@Solace.com>.
Figured it out. Basically you cannot log inside the getSplitBacklogBytes() method and get it to show in the UnboundedSource.Read log stack and could not figure out another way to get at this logs. So I added stats in this method and dumped the stats periodically in the advance() method to get an indecation on what was going on. My backlog method was returning 0 which was holding down auto-scaling. Basically everything was working as designed, just difficult to troubleshoot.
On 2019/01/09 20:09:46, Ken Barr <Ke...@solace.com> wrote:
> Hello
>
> I have been directed to this mail list by posting to Stack Overflow. If this is inappropriate use of mail list please let me know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
>
> I am creating a custom IO connector in Java that runs is streaming mode with 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, similar to JMS IO. All works well except the auto-scaling does not seem to work. The odd part for me is that I have implemented a getSplitBacklogBytes() method, but as far as the logs show, it is never executed.
>
> I am testing with a modified wordcount example to test, executed with:
>
> --runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=4
>
>
>
> The Pipeline options and logs seem to show that auto-scaling is being set up correctly.
>
> PipelineOptions: Current Settings:
>
> autoscalingAlgorithm: THROUGHPUT_BASED
>
> maxNumWorkers: 4
>
> numWorkers: 0
>
> Logs show:
>
> 20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will be between 1 and 4.
>
> 20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker processes.
>
> 20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent disks which will allow autoscaling from 1 to 4 workers.
>
> 20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers...
>
>
>
> Is there anything else that needs to be done to enable this feature, or anything I might have done to inadvertently disable this feature?
>
> Source code lives here: https://github.com/SolaceLabs/solace-beam-integration/tree/development
>
>