You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/10/11 14:01:33 UTC

[GitHub] [flink] mxm opened a new pull request, #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

mxm opened a new pull request, #21023:
URL: https://github.com/apache/flink/pull/21023

## What is the purpose of the change

This allows to change the job vertex parallelism of a JobGraph during job submission time without having to modify the JobGraph upfront.

The initial idea was to add a new field to the payload of the job submit Rest endpoint. However, it is probably more practical to handle the overrides in the same way as other PipelineOptions already do it, i.e. via the configuration.

The implementation is deliberately lenient with respect to the presence of job vertices. If vertices have been removed or new ones have been added, only the ones found will have their parallelism overrides. The verification should be performed by the caller, not by Flink. In particular, we want to support scenarios where users modify the deployment and we might not yet have overrides for all vertices.

Based on feedback in https://github.com/apache/flink/pull/20953.

## Brief change log

- Add option to override job vertex parallelisms during job submission
- Allow Map<String, T> ConfigOptions

## Verifying this change

Tests

## Does this pull request potentially affect one of the following parts:

- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
- The S3 file system connector: (no)

## Documentation

- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] mxm commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

mxm commented on PR #21023:
URL: https://github.com/apache/flink/pull/21023#issuecomment-1311934193

   I've removed the dependency on #21008.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] mxm commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

mxm commented on PR #21023:
URL: https://github.com/apache/flink/pull/21023#issuecomment-1317226026

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] gyfora commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

gyfora commented on PR #21023:
URL: https://github.com/apache/flink/pull/21023#issuecomment-1313515057

   Seems like the doc need to be regenerated:
   ```
   Option pipeline.jobvertex-parallelism-overrides in class org.apache.flink.configuration.PipelineOptions is not documented.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

flinkbot commented on PR #21023:
URL: https://github.com/apache/flink/pull/21023#issuecomment-1274753160

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "734a5d464da2ced755f0f71492d037f666c78313",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "734a5d464da2ced755f0f71492d037f666c78313",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 734a5d464da2ced755f0f71492d037f666c78313 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] mxm commented on pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

mxm commented on PR #21023:
URL: https://github.com/apache/flink/pull/21023#issuecomment-1311937328

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] mxm merged pull request #21023: [FLINK-29501] Add option to override job vertex parallelisms during job submission

Posted by GitBox <gi...@apache.org>.

mxm merged PR #21023:
URL: https://github.com/apache/flink/pull/21023


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org