You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Edwin <xu...@163.com> on 2022/03/01 11:30:33 UTC

Adaptive Batch Configuration Does Not Work

Hi all,


I was trying to run flink-tpcds tests on flink v1.15-snapshot, and I added "jobmanager.adaptive-batch-scheduler.default-source-parallelism: 8" along with some other configurations to the conf yaml file to enable adaptive job scheduler. But it turned out the source parallelism kept unchanged(always be 1), I was wondering if anyone could kindly give some advice on how to fix this? some of my configurations are listed below:


jobmanager.scheduler: AdaptiveBatch
jobmanager.adaptive-batch-scheduler.max-parallelism: 64
jobmanager.adaptive-batch-scheduler.data-volume-per-task: 6m
jobmanager.adaptive-batch-scheduler.default-source-parallelism: 8


metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: true


parallelism.default: -1


Thanks!

Re: Adaptive Batch Configuration Does Not Work

Posted by Lijie Wang <wa...@gmail.com>.

Thanks for feedback! Can you post the jobmanager log? This may be because
the source parallelism has been set in some way (Kind of source can infer
parallelism according to the catalog).

Best,
Lijie

Edwin <xu...@163.com> 于2022年3月1日周二 19:30写道：

> Hi all,
>
> I was trying to run flink-tpcds tests on flink v1.15-snapshot, and I added
> "jobmanager.adaptive-batch-scheduler.default-source-parallelism: 8" along
> with some other configurations to the conf yaml file to enable adaptive job
> scheduler. But it turned out the source parallelism kept unchanged(always
> be 1), I was wondering if anyone could kindly give some advice on how to
> fix this? some of my configurations are listed below:
>
> jobmanager.scheduler: AdaptiveBatch
> jobmanager.adaptive-batch-scheduler.max-parallelism: 64
> jobmanager.adaptive-batch-scheduler.data-volume-per-task: 6m
> jobmanager.adaptive-batch-scheduler.default-source-parallelism: 8
>
> metrics.reporter.promgateway.class:
> org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
> metrics.reporter.promgateway.randomJobNameSuffix: true
> metrics.reporter.promgateway.deleteOnShutdown: true
>
> parallelism.default: -1
>
> Thanks!
>
>
>
>
>