You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zjureel <gi...@git.apache.org> on 2017/05/16 10:42:50 UTC
[GitHub] flink pull request #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from Con...
GitHub user zjureel opened a pull request:
https://github.com/apache/flink/pull/3921
[FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnvironment
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjureel/flink FLINK-6058
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3921.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3921
----
commit 808f252f64ac96569a16ba3d0a020ad978983513
Author: zjureel <zj...@gmail.com>
Date: 2017-05-16T07:58:10Z
fix read DEFAULT_PARALLELISM from ContextEnvironment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/3921
Thanks! 👍 I merged this, could you please close the PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/3921
I think we can get by without changing `ClusterClient` and `ContextEnvironment` by only reading the parallelism from the global config in `CliFrontend` when trying to read the user parallelism from the parameters, i.e. in `run()` and `info()`. In the example of `run()` we could change this code:
```
int userParallelism = options.getParallelism();
LOG.debug("User parallelism is set to {}", userParallelism);
if (client.getMaxSlots() != -1 && userParallelism == -1) {
logAndSysout("Using the parallelism provided by the remote cluster ("
+ client.getMaxSlots() + "). "
+ "To use another parallelism, set it at the ./bin/flink client.");
userParallelism = client.getMaxSlots();
}
return executeProgram(program, client, userParallelism);
```
to this
```
int parallelism = options.getParallelism();
LOG.debug("User parallelism is set to {}", parallelism);
if (client.getMaxSlots() != -1 && parallelism == -1) {
logAndSysout("Using the parallelism provided by the remote cluster ("
+ client.getMaxSlots() + "). "
+ "To use another parallelism, set it at the ./bin/flink client.");
parallelism = client.getMaxSlots();
} else if (parallelism == ExecutionConfig.PARALLELISM_DEFAULT) {
parallelism = GlobalConfiguration.loadConfiguration().getInteger(
ConfigConstants.DEFAULT_PARALLELISM_KEY,
ConfigConstants.DEFAULT_PARALLELISM);
}
return executeProgram(program, client, parallelism);
```
with this change `StreamContextEnvironment` would simply need this:
```
if (ctx.getParallelism() > 0) {
setParallelism(ctx.getParallelism());
}
```
because the environment will have the default parallelism set (in `ContextEnvironmentFactory` https://github.com/apache/flink/blob/c793ea41d88fe84fa97d825728ad95f35e27ef82/flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironmentFactory.java#L52-L52)
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/3921
I have updated the code according to your suggestion, please have a look when you are free, thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/3921
Thank you for merging it @aljoscha
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3921
According to the JIRA neither environment should access the global configuration, but instead either pass it through a setter or constructor argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from Con...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel closed the pull request at:
https://github.com/apache/flink/pull/3921
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/3921
I have fixed it, thanks @aljoscha
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/3921
@aljoscha Sorry for replying late, your suggestion sounds good to me. In fact, I was hesitating to add `getDefaultParallelism()` in `ContextEnvironment` and `ClusterClient` for there have been `getParallelism()` in `ContextEnvironment`. Thank you for your suggestion, I will fix it as you suggested soon :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/3921
@zjureel It seems `CliFrontendRunTest` fails because the expected parallelism is not `-1` anymore https://travis-ci.org/apache/flink/jobs/251014376. After fixing this, let's see if any other tests fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3921: [FLINK-6058] fix read DEFAULT_PARALLELISM from ContextEnv...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/3921
@zentol Thank you for your comment, I have fixed the problem you metioned. Could you please review the new commit when you are free? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---