You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2014/01/09 18:50:55 UTC
[jira] [Comment Edited] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866845#comment-13866845 ]
Alejandro Abdelnur edited comment on YARN-888 at 1/9/14 5:50 PM:
-----------------------------------------------------------------
[~vinodkv],
While having dependencies in non-leaf POMs may reduce the size of leaf POMs, it drags non-required dependencies (unless you only put in non-leaf POMs dependencies that are common to all the leaf modules).
Yes IntelliJ seems to get funny with dependencies in non-leaf modules. That is one of the motivations (agree it is an IntelliJ issue, on the other hand the change does not affect the project built at all and allows IntelliJ users to build/debug from the IDE out of the box without doing funny voodoo).
The other motivation, and IMO is more important, is to clean up the dependencies modules like yarn-api and yarn-client have. Restricting them to what is used on the client side. Using the dependency:tree and dependency:analyze plugins I’ve reduced the 3rd party JARs required by the clients significantly. As [~stevel@apache.org] pointed out there is much more work we should do in this direction, this is a first non-intrusive baby step in that direction.
To give you and idea, before the this patch *hadoop-yarn-api* reports as required dependencies by itself:
{code}
+- org.slf4j:slf4j-api:jar:1.7.5:compile
+- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
| \- log4j:log4j:jar:1.2.17:compile
+- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
| +- tomcat:jasper-compiler:jar:5.5.23:test
+- com.google.inject.extensions:guice-servlet:jar:3.0:compile
+- io.netty:netty:jar:3.6.2.Final:compile
+- com.google.protobuf:protobuf-java:jar:2.5.0:compile
+- commons-io:commons-io:jar:2.4:compile
+- com.google.inject:guice:jar:3.0:compile
| +- javax.inject:javax.inject:jar:1:compile
| \- aopalliance:aopalliance:jar:1.0:compile
+- com.sun.jersey:jersey-server:jar:1.9:compile
| +- asm:asm:jar:3.2:compile
| \- com.sun.jersey:jersey-core:jar:1.9:compile
+- com.sun.jersey:jersey-json:jar:1.9:compile
| +- org.codehaus.jettison:jettison:jar:1.1:compile
| | \- stax:stax-api:jar:1.0.1:compile
| +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
| | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
| | \- javax.activation:activation:jar:1.1:compile
| +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
| +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
| +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile (version managed from 1.8.3)
| \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile (version managed from 1.8.3)
\- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
{code}
With the patch, the required dependencies by itself are down to:
{code}
+- commons-lang:commons-lang:jar:2.6:compile
+- com.google.guava:guava:jar:11.0.2:compile
| \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
+- commons-logging:commons-logging:jar:1.1.3:compile
+- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
\- com.google.protobuf:protobuf-java:jar:2.5.0:compile
{code}
Does this address your concerns?
was (Author: tucu00):
[~vinodvk],
While having dependencies in non-leaf POMs may reduce the size of leaf POMs, it drags non-required dependencies (unless you only put in non-leaf POMs dependencies that are common to all the leaf modules).
Yes IntelliJ seems to get funny with dependencies in non-leaf modules. That is one of the motivations (agree it is an IntelliJ issue, on the other hand the change does not affect the project built at all and allows IntelliJ users to build/debug from the IDE out of the box without doing funny voodoo).
The other motivation, and IMO is more important, is to clean up the dependencies modules like yarn-api and yarn-client have. Restricting them to what is used on the client side. Using the dependency:tree and dependency:analyze plugins I’ve reduced the 3rd party JARs required by the clients significantly. As [~stevel@apache.org] pointed out there is much more work we should do in this direction, this is a first non-intrusive baby step in that direction.
To give you and idea, before the this patch *hadoop-yarn-api* reports as required dependencies by itself:
{code}
+- org.slf4j:slf4j-api:jar:1.7.5:compile
+- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
| \- log4j:log4j:jar:1.2.17:compile
+- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
| +- tomcat:jasper-compiler:jar:5.5.23:test
+- com.google.inject.extensions:guice-servlet:jar:3.0:compile
+- io.netty:netty:jar:3.6.2.Final:compile
+- com.google.protobuf:protobuf-java:jar:2.5.0:compile
+- commons-io:commons-io:jar:2.4:compile
+- com.google.inject:guice:jar:3.0:compile
| +- javax.inject:javax.inject:jar:1:compile
| \- aopalliance:aopalliance:jar:1.0:compile
+- com.sun.jersey:jersey-server:jar:1.9:compile
| +- asm:asm:jar:3.2:compile
| \- com.sun.jersey:jersey-core:jar:1.9:compile
+- com.sun.jersey:jersey-json:jar:1.9:compile
| +- org.codehaus.jettison:jettison:jar:1.1:compile
| | \- stax:stax-api:jar:1.0.1:compile
| +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
| | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
| | \- javax.activation:activation:jar:1.1:compile
| +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
| +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
| +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile (version managed from 1.8.3)
| \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile (version managed from 1.8.3)
\- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
{code}
With the patch, the required dependencies by itself are down to:
{code}
+- commons-lang:commons-lang:jar:2.6:compile
+- com.google.guava:guava:jar:11.0.2:compile
| \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
+- commons-logging:commons-logging:jar:1.1.3:compile
+- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
\- com.google.protobuf:protobuf-java:jar:2.5.0:compile
{code}
Does this address your concerns?
> clean up POM dependencies
> -------------------------
>
> Key: YARN-888
> URL: https://issues.apache.org/jira/browse/YARN-888
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.1.0-beta
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Attachments: YARN-888.patch, YARN-888.patch, yarn-888-2.patch
>
>
> Intermediate 'pom' modules define dependencies inherited by leaf modules.
> This is causing issues in intellij IDE.
> We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)