You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2017/06/13 21:34:56 UTC
[GitHub] flink pull request #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedu...
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4120
[FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCase to work with Hadoop 2.6.5, 2.7.3 and 2.8.0
Due to MNG-5899, maven cannot resolve dependency reduced poms in a multi project build. Therefore, flink-yarn-tests pulls in a wrong version of org.apache.httpcomponents.httpclient which does not work with Hadoop's ServletUtils together. As a solution we have to move the dependency management for the httpclient and httpcore version into the parent pom.xml.
Another problem is the version of these libraries which has been recently bumped. In 4.4, httpclient changed its behaviour such that URLEncodedUtils#parse(String, Charset) now throws a NPE if the first parameter is null. In 4.2.6, an empty list was returned instead. Due to this incompatibility, we reverted the change and set the version to its previous value.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixYarnSessionTest
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4120
----
commit 37d5577cd12454ee3b5aa39206989bb89faf0cf8
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-06-13T21:26:10Z
[FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCase to work with Hadoop 2.6.5, 2.7.3 and 2.8.0
Due to MNG-5899, maven cannot resolve dependency reduced poms in a multi project build. Therefore,
flink-yarn-tests pulls in a wrong version of org.apache.httpcomponents.httpclient which does not work
with Hadoop's ServletUtils together. As a solution we have to move the dependency management for the
httpclient and httpcore version into the parent pom.xml.
Another problem is the version of these libraries which has been recently bumped. In 4.4, httpclient
changed its behaviour such that URLEncodedUtils#parse(String, Charset) now throws a NPE if the first
parameter is null. In 4.2.6, an empty list was returned instead. Due to this incompatibility, we reverted
the change and set the version to its previous value.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCa...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4120
The PR descriptions says that you're downgrading the dependency versions, but with the second commit your actually upgrading them.
Also, couldn't we move the dependency management into `flink-shaded-hadoop`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCa...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4120
Rebasing onto the latest master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCa...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4120
I first thought I had to downgrade, because of the aforementioned problem. However, Hadoop 2.8.0 needs at least httpclient `4.5.2`. Luckily, they fixed the problem with the `URLEncodedUtils#parse` method in `4.5.3`. Therefore I bumped the version again.
Concerning moving the dependency management to `flink-shaded-hadoop`: I think this is unfortunately not possible since due the immutable reactor builds from maven, `flink-yarn-tests` pulls all dependencies from `flink-shaded-hadoop` and `flink-shaded-yarn-tests`. This also means that depending on the classpath order, we either use the shaded Yarn mini-cluster or not. In the latter case we have to make sure that we are using `httpclient` `4.5.3` which is only the case if we define the dependency management in the parent pom.
As a side remark, given this situation with maven, I'm not sure whether we actually need the `flink-shaded-yarn-tests` module.
A proper solution would be fixing the problem with the immutable builds. For example, everything works properly if we first install the shaded jars and then build the flink-yarn-tests. Then maven will properly resolve all dependencies. In this case, I think it should be possible to move the dependency management into `flink-shaded-hadoop`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCa...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4120
Thanks for the review @zentol. Failing test case is unrelated and fixed via #4139. Merging this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedulerITCa...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4120
+1 to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4120: [FLINK-6836] [tests] Fix YARNSessionCapacitySchedu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---