You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by yew1eb <gi...@git.apache.org> on 2018/06/04 16:12:16 UTC
[GitHub] flink pull request #6116: [FLINK-9498][build] Disable dependency convergence...
GitHub user yew1eb opened a pull request:
https://github.com/apache/flink/pull/6116
[FLINK-9498][build] Disable dependency convergence for flink-end-to-end-tests
## What is the purpose of the change
*deactivate that enforce dependency convergence for `flink-end-to-end-tests` module*
## Brief change log
- *skip enforce dependency convergence in the `flink-end-to-end-tests` module pom*
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yew1eb/flink FLINK-9498
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6116.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6116
----
commit e564d752decf24edf9ae7163f65d83fede0c2567
Author: yew1eb <ye...@...>
Date: 2018-06-04T14:00:07Z
[FLINK-9498][build] Disable dependency convergence for flink-end-to-end-tests
----
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
Why make this change? I don't see a build failure; i guess this is just another instance of FLINK-9091 in which case the PR is already subsumed by #6102.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/6116
Will it lower the possibility of detecting lib version conflicts of Flink's dependencies?
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/6116
Yes I know, but I didn't see those problematic dependencies in
> mvn -Dhadoop.version=2.7.0 dependency:tree -pl flink-shaded-hadoop/flink-shaded-hadoop2
thus I why assumed problem is with `2.7.0-xxx`.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by yew1eb <gi...@git.apache.org>.
Github user yew1eb commented on the issue:
https://github.com/apache/flink/pull/6116
@pnowojski yes, some of dependencis in the hadoop version maintained by our company have some coflicts with flink.
do you have any suggestions for me?
---
[GitHub] flink pull request #6116: [FLINK-9498][build] Disable dependency convergence...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/6116
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
ah sorry, i didn't see `-xxx`.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/6116
I think the problem here is not that some of your versions are conflicting with flink, but that your dependencies are on their own conflicting. When I check:
> mvn -Dhadoop.version=2.7.0 dependency:tree -pl flink-shaded-hadoop/flink-shaded-hadoop2
I do not see any `libthrift` references, and only `commons-lang` comes from hadoop itself. You should probably ask the guys maintaining your hadoop `2.7.0-xxx` to converge those dependencies (for example pin `commons-lang` to 2.6 and `libthrift` to 0.9), otherwise bad things can happen randomly.
I'm not sure how could the solution on Flink side look like. @zentol maybe as we discussed previously, we could provide build profiles for some set of supported by us hadoop versions. But is there a mechanism for user to provide a custom build profile for his own custom hadoop version that solves the convergence in `flink-shaded-hadoop2`?
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
dependency-convergence is now disabled by default and only checked on travis, or manually using the `tools/check-dependency-convergency.sh` script.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/6116
Or can not you converge dependencies in your hadoop? It seems like it's causing convergence errors with itself.
Disabling convergence checking would be a step back and asking ourself for even more trouble in the future when debugging deadlocks or other random errors.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
The actual issue here is that these dependencies aren't shaded.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
Well it's not really _their_ hadoop, we explicitly allow and intend users to set different hadoop version for `flink-shaded-hadoop`. For the default version in master we're hiding all dependencies, but we don't cover other versions. This also means that enabling dependency convergence for flink-shaded-hadoop is effectively really really difficult since it would have to apply to different versions....
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/6116
I'm not aware of a mechanism that allows defining an adhoc dependency convergence. Users will have to modify the pom of `flink-shaded-hadoop` and add a convergence profile for the given version, similar to what we discussed.
In any case, I think this PR can be closed.
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6116
Is there a workaround for users to disable dependency convergence?
It is actually a problem that we don't control the convergence of some dependency that is used with varying versions (Hadoop), but rely on it for our own builds to succeed.
In the long run, we may want to work only "hadoop free", referencing for compilation-only our own shaded Hadoop (maybe even move it to flink-shaded) and let users export the Hadoop Classpath for their own Hadoop (rather than changing the dependency when building Flink).
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by yew1eb <gi...@git.apache.org>.
Github user yew1eb commented on the issue:
https://github.com/apache/flink/pull/6116
CC @pnowojski
---
[GitHub] flink issue #6116: [FLINK-9498][build] Disable dependency convergence for fl...
Posted by yew1eb <gi...@git.apache.org>.
Github user yew1eb commented on the issue:
https://github.com/apache/flink/pull/6116
when I build flink with our maintained hadoop version, the command like this `mvn clean install -DskipTests -Dhadoop.version=2.7.0-xxx`, it will `BUILD FAILURE`.
Some Enforcer rules have failed. Look below for specific messages explaining why the rule failed.
```
[WARNING]
Dependency convergence error for commons-lang:commons-lang:2.6 paths to dependency are:
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-commons-lang:commons-lang:2.6
and
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-commons-configuration:commons-configuration:1.7
+-commons-lang:commons-lang:2.6
and
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-org.apache.thrift:libthrift:0.9.0
+-commons-lang:commons-lang:2.5
[WARNING]
Dependency convergence error for org.apache.thrift:libthrift:0.9.0 paths to dependency are:
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-org.apache.thrift:libthrift:0.9.0
and
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-com.xxxx.scribe:scribe-log4j:1.1.0
+-org.apache.thrift:libthrift:0.8.0
and
+-org.apache.flink:flink-bucketing-sink-test:1.6-SNAPSHOT
+-org.apache.flink:flink-shaded-hadoop2:1.6-SNAPSHOT
+-com.xxxx.log:collect-client:0.0.5
+-org.apache.thrift:libthrift:0.8.0
```
---