You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2017/05/09 13:54:18 UTC

[GitHub] flink pull request #3856: [FLINK-5998] Fix shaded Hadoop Jar

GitHub user aljoscha opened a pull request:

    https://github.com/apache/flink/pull/3856

    [FLINK-5998] Fix shaded Hadoop Jar

    Now that flink-dist has the shaded hadoop jar as a provided dependency
    we need to ensure that the shaded hadoop jar includes all the
    dependencies that it needs because flink-dist does not include them
    anymore in its shading.
    
    The manifestation of this was that we couldn't start a cluster anymore
    because of missing class files.
    
    The upside of this PR is that we can run a cluster again. The downside is that we now have some dependencies both in the `flink-dist` jar and in the shaded Hadoop jar, i.e. total size increases.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink fix-shaded-hadoop

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3856
    
----
commit ce0eb550277b9dd737f4dabb4eb586d36bd6ed35
Author: Aljoscha Krettek <al...@gmail.com>
Date:   2017-05-09T13:50:09Z

    [FLINK-5998] Fix shaded Hadoop Jar
    
    Now that flink-dist has the shaded hadoop jar as a provided dependency
    we need to ensure that the shaded hadoop jar includes all the
    dependencies that it needs because flink-dist does not include them
    anymore in its shading.
    
    The manifestation of this was that we couldn't start a cluster anymore
    because of missing class files.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by rmetzger <gi...@git.apache.org>.

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3856

I think the fix works in almost all cases.
There's only one problem: Your change causes the `flink-shaded-hadoop2` artifact on maven central to not expose any dependencies anymore. So a module referencing this (like `flink-java`) will not see what `flink-shaded-hadoop2` contains. This can cause problems like having classes in the classpath multiple times. Maven can not "manage" the dependencies in that case, because it does not know what's `flink-shaded-hadoop2`.

I don't have a good answer how to solve this.
Some ideas:
- relocate all Hadoop dependencies in the `flink-shaded-hadoop2` artifact. Then, we won't run into the original problem anymore. I've tried this once, but I was running into problems getting the YARN tests running afterwards
- introduce a special "flink-shaded-hadoop2-dist" module that prepares a fat dist hadoop jar for the binary distribution. This way, we can differentiate between `flink-shaded-hadoop2` as a dependency and for the binary. But I think this will lead to problems when building `flink-dist`...
- merge this PR as is and hope that the problems don't occur (I think this is mostly relevant for people using Hadoop code in their user jar, for example when doing some Hadoop compatibility stuff)

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by gyfora <gi...@git.apache.org>.

Github user gyfora commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    This fix seems to work for me as well 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    R: @rmetzger Could you please have a quick look, if this makes sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    @gyfora See https://issues.apache.org/jira/browse/FLINK-6514, it seems that this only occurs with newer Maven versions. When building with 3.2.5 it works.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by gyfora <gi...@git.apache.org>.

Github user gyfora commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    @aljoscha Yes, that might work although I prefer this fix compared to having to mess with the maven versions on deploy servers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    Superseded


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar

Posted by rmetzger <gi...@git.apache.org>.

Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/3856
  
    I think Stephan is working on fixing this properly for all Maven versions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---