You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@flink.apache.org by Azure DevOps <az...@microsoft.com.INVALID> on 2021/03/22 20:08:59 UTC
[PR build failed] flink-ci.flink-master-mirror -
[FLINK-21844][runtime] Do not auto-configure maxParallelism in REACTIVE
scheduling mode - flink-ci...
[PR build failed] flink-ci.flink-master-mirror - [FLINK-21844][runtime] Do not
auto-configure maxParallelism in REACTIVE scheduling mode - flink-ci/flink-
mirror:FLINK-21844 - apache-flink - e1e5cc46
| | | |
 |
Azure DevOps
---|---
| | | |  | Build #20210322.7 failed
---|---
flink-ci.flink-master-mirror
Ran for 8 minutes
| | [ View results ](https://dev.azure.com/apache-
flink/web/build.aspx?pcguid=2d3c0ac8-fecf-45be-8407-6d87302181a9&builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f15211&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzNzk4NDMiLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NjkyMzczNjcsIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMmMzMjQyYjgtZjVlNC00Zjc0LTgxYWUtZGNjZTJkODMzYWUwIn0sIkVsZW1lbnQiOiJoZXJvL2N0YSJ9)
---
| | | | Pull request
---
| Title | [[FLINK-21844][runtime] Do not auto-configure maxParallelism in
REACTIVE scheduling mode](https://github.com/flink-ci/flink-mirror/pull/7)
---|---
Source branch | FLINK-21844
Target branch | master
Description |
## What is the purpose of the change
To maintain compatibility between reactive and default scheduling modes, the
maxParallelism of operators must be set consistently. Currently, there are
different defaults applied depending on the mode. This PR unifies setting
those defaults for JobGraphs to the jobgraph package via a util class. I also
considered adding the defaulting logic to the JobGraphBuilder, but that seems
to be mostly used for tests.
The main change is bringing the defaulting logic out of the
ExecutionJobVertex, and then allowing the JobVertex to know if its
maxParallelism been explicitly set by the user or the system. I've also tried
to consolidate the number of places where the -1 default magic number is
defined.
## Brief change log
* Both reactive and default JobGraphs are configured in the DefaultJobManagerRunnerFactory
* JobVertex has a package-private method for setting the auto-configured flag, which is used when restoring from operator state among other things
* Tests updated to use the common helper methods
## Verifying this change
This change added tests and can be verified as follows:
* Refactored current auto-configuration test into a parameterized JobGraphConfigurationUtilsAutoConfigurtionTest that tests both default and reactive parallelism auto-configuration
* Updated existing to explicitly add parallelism when creating vertices
## Does this pull request potentially affect one of the following parts:
* Dependencies (does it add or upgrade a dependency): no
* The public API, i.e., is any changed class annotated with @Public(Evolving): no
* The serializers: no
* The runtime per-record code paths (performance sensitive): no
* Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes
* The S3 file system connector: no
## Documentation
* Does this pull request introduce a new feature? no
* If yes, how is the feature documented? JavaDocs
| | Summary
---
| Build pipeline | flink-ci.flink-master-mirror
---|---
Finished | Mon, Mar 22 2021 20:08:37 GMT+00:00
Requested for | GitHub
Reason | Private
| | Details
---
| e2e_cron_hadoop241
---
| 0 error(s), 0 warning(s)
---
compile_cron_hadoop241
| 0 error(s), 0 warning(s)
---
e2e_ci
| 1 error(s), 0 warning(s)
---
 | Bash
exited with code '1'.
compile_cron_python_wheels
| 0 error(s), 0 warning(s)
---
test_cron_scala212
| 0 error(s), 0 warning(s)
---
cron_snapshot_deployment_maven
| 0 error(s), 0 warning(s)
---
compile_ci
| 1 error(s), 0 warning(s)
---
 | Bash
exited with code '1'.
compile_cron_azure
| 0 error(s), 0 warning(s)
---
docs_404_check
| 0 error(s), 0 warning(s)
---
build_wheels
| 0 error(s), 0 warning(s)
---
e2e_cron_jdk11
| 0 error(s), 0 warning(s)
---
cron_snapshot_deployment_binary
| 0 error(s), 0 warning(s)
---
docs_404_check
| 0 error(s), 0 warning(s)
---
test_cron_adaptive_scheduler
| 0 error(s), 0 warning(s)
---
e2e_cron_hadoop313
| 0 error(s), 0 warning(s)
---
test_cron_jdk11
| 0 error(s), 0 warning(s)
---
test_cron_hadoop241
| 0 error(s), 0 warning(s)
---
compile_cron_scala212
| 0 error(s), 0 warning(s)
---
compile_cron_adaptive_scheduler
| 0 error(s), 0 warning(s)
---
test_cron_azure
| 0 error(s), 0 warning(s)
---
compile_cron_jdk11
| 0 error(s), 0 warning(s)
---
e2e_cron_scala212
| 0 error(s), 0 warning(s)
---
compile_cron_hadoop313
| 0 error(s), 0 warning(s)
---
test_cron_hadoop313
| 0 error(s), 0 warning(s)
---
e2e_cron_azure
| 0 error(s), 0 warning(s)
---
e2e_cron_adaptive_scheduler
| 0 error(s), 0 warning(s)
---
test_ci
| 0 error(s), 0 warning(s)
---
| | Commits
---
| [a24f5a0a](https://github.com/flink-ci/flink-
mirror/commit/a24f5a0a4c5de215b632ffe457333aa840972f72) | [FLINK-21844] Do
not auto-configure maxParallelism in REACTIVE scheduling mode | austince
---|---|---
[3657c1a5](https://github.com/flink-ci/flink-
mirror/commit/3657c1a5a886b894138643d96098c2e5ba15120d) | [hotfix][runtime]
Fix typo in DefaultExecutionGraph method parameter | austince
| We sent you this notification on behalf of [apache-flink]\apache-flink
Team. [View](https://dev.azure.com/apache-
flink/_notifications?subscriptionId=379843&publisherId=ms.vss-build.build-
event-
publisher&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzNzk4NDMiLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NjkyMzczNjcsIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMmMzMjQyYjgtZjVlNC00Zjc0LTgxYWUtZGNjZTJkODMzYWUwIn0sIkVsZW1lbnQiOiJmb290ZXIvdmlld0xpbmsifQ%3d%3d)
---
Microsoft respects your privacy. Review our Online Services [Privacy
Statement](https://privacy.microsoft.com/privacystatement).
One Microsoft Way, Redmond, WA, USA 98052.
Sent from Azure DevOps
