[jira] [Created] (FLINK-26548) the source parallelism is not set correctly with AdaptiveBatchScheduler

["zl (Jira)" <ji...@apache.org>]

zl created FLINK-26548:
--------------------------

             Summary: the source parallelism is not set correctly with AdaptiveBatchScheduler
                 Key: FLINK-26548
                 URL: https://issues.apache.org/jira/browse/FLINK-26548
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Task
    Affects Versions: 1.15.0
            Reporter: zl
         Attachments: image-2022-03-09-19-00-18-396.png

When running *_org.apache.flink.table.tpcds.TpcdsTestProgram_* with {_}*AdaptiveBatchScheduler*{_}, I ran into a problem：the num of records sent by the source operator is always 1, and the parallelism of source operator is also 1 even I set *_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* to 8.

!image-2022-03-09-19-00-18-396.png!

After some research, I found that the operator A is not the actual file reader, it just splits files and assigns splits to downstream tasks for further processing, and the operator B is the actual file reader task. Here, the parallelism of operator B is 64, and the records sent by operator A is 1, this means, operator A assigned all splits to a task of operator B, {*}_the other 63 tasks of operator B is idle_{*}, it is unreasonable.

In this case,  the parallelism of operator B should be *_jobmanager.adaptive-batch-scheduler.default-source-parallelism_*  and the num of records sent by operator A also should be {*}_jobmanager.adaptive-batch-scheduler.default-source-parallelism_{*}.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)