You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/22 11:32:00 UTC
[jira] [Commented] (FLINK-9289) Parallelism of generated operators
should have max parallism of input
[ https://issues.apache.org/jira/browse/FLINK-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520267#comment-16520267 ]
ASF GitHub Bot commented on FLINK-9289:
---------------------------------------
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/6203
[FLINK-9289][rest] Rework JobSubmitHandler to accept jar/artifact files
Builds on #6199.
## What is the purpose of the change
This PR reworks the `JobSubmitHandler` to accept jars/artifacts and the jobgraph in a single REST request, and changes the `RestClusterClient` to make use of this feature.
As a result the entire job submission now goes through the REST API.
There are still known issues in this PR but they won't impact the correctness; the core implementation is complete, works and is covered by tests.
## Brief change log
* remove blobserver port headers/handlers and usages
* rework `JobSubmitRequestBody` to contain the names of the jobgraph, jar and artifact files, providing semantics to the collection of uploaded files
* set visibility of utility methods in `ClientUtils` to `public`to allow passing jar/artifact collections manually
* modified the `JobSubmitHandler` to
* read the jobgraph from an uploaded file
* upload uploaded jars/artifacts to the blob service and store keys in the jobgraph
* modified the `RestClusterClient` to
* write jobgraph into a file for later upload
* make use of new `RestClient` API for multipart requests
* extended ´RestConstants` for jar/binary `content-types`, and made the class an enum to get singleton properties
* extended documentation of `JobSubmitHeaders`
## Verifying this change
* this PR only modifies existing tests to accommodate changes
* the general functionality is covered by existing end-to-end tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 9280_epsilon
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6203.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6203
----
commit 90d81f7bdde62f2b9bb1d1da071cc3d0b0d1b349
Author: zentol <ch...@...>
Date: 2018-06-13T16:21:21Z
[FLINK-9624][runtime] Move jar/artifact upload out of jobgraph
commit a365697a031d16a778f25334c3bc2c7706d13693
Author: zentol <ch...@...>
Date: 2018-06-11T09:45:12Z
[FLINK-9280][rest] Remove BlobServer port handler
commit 78b456eb861fd722a100ae6e153f1c0079649d8b
Author: zentol <ch...@...>
Date: 2018-06-22T09:03:45Z
[FLINK-9289][rest] Rework JobSubmitHandler to accept jar/artifact files
----
> Parallelism of generated operators should have max parallism of input
> ---------------------------------------------------------------------
>
> Key: FLINK-9289
> URL: https://issues.apache.org/jira/browse/FLINK-9289
> Project: Flink
> Issue Type: Bug
> Components: DataSet API
> Affects Versions: 1.5.0, 1.4.2, 1.6.0
> Reporter: Fabian Hueske
> Assignee: Xingcan Cui
> Priority: Major
> Labels: pull-request-available
>
> The DataSet API aims to chain generated operators such as key extraction mappers to their predecessor. This is done by assigning the same parallelism as the input operator.
> If a generated operator has more than two inputs, the operator cannot be chained anymore and the operator is generated with default parallelism. This can lead to a {code}NoResourceAvailableException: Not enough free slots available to run the job.{code} as reported by a user on the mailing list: https://lists.apache.org/thread.html/60a8bffcce54717b6273bf3de0f43f1940fbb711590f4b90cd666c9a@%3Cuser.flink.apache.org%3E
> I suggest to set the parallelism of a generated operator to the max parallelism of all of its inputs to fix this problem.
> Until the problem is fixed, a workaround is to set the default parallelism at the {{ExecutionEnvironment}}:
> {code}
> ExecutionEnvironment env = ...
> env.setParallelism(2);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)