You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID> on 2016/06/20 16:18:48 UTC
Shading Plugin + Hadoop Dependency
Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class.
However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package.
Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components?
mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase
[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
RegardsVijay
Re: Shading Plugin + Hadoop Dependency
Posted by vijikarthi <vi...@yahoo.com>.
Any modifications to "flink-shaded-include-yarn-tests" with respect to the
dependency inclusion/exclusion does not get reflected in the
"flink-yarn-tests" test case since the referenced packages from
"flink-shaded-include-yarn-tests" are not being used (though the pom file
establish the dependency). I am able to confirm by deleting the dependency
chain from the "flink-shaded-include-yarn-tests" and rerun the
"flink-yarn-tests" IT test cases. The reason why this works because of the
reference to the "flink-shaded-includeyarn-tests" packages from "local"
maven repository being in the classpath.
I would like to understand if this has been designed like this or is it a
bug? I am not able to succesfully include a new package (hadoop-minikdc) and
write new test cases to run "Yarn" test module. Any help is highly
appreciated.
I use below commands to run the "test" module and in case if that is
incorrect please let me know the right way to run the tests.
mvn clean package (to build the source and run unit test)
mvn clean verify -pl flink-yarn-tests -Pinclude-yarn-tests (to run Yarn
test)
I also use the filter ( -Dtest=YARNSessionFIFOITCase#testJavaAPI ) to run
specific test case.
P.S: I have also noticed the surefile plugin is not stable and sometimes it
throws some wierd issues which is very difficult to debug.
Regards
Vijay
--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Shading-Plugin-Hadoop-Dependency-tp12174p12232.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.
Re: Shading Plugin + Hadoop Dependency
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Hi Rob,
I ran the build and now I could see the minikdc classes bundled in "flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar" location.
cd /workspace/git-projects/flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests
$ jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdcminikdc-krb5.confminikdc.ldifforg/apache/hadoop/minikdc/org/apache/hadoop/minikdc/KerberosSecurityTestcase.classorg/apache/hadoop/minikdc/MiniKdc$1.classorg/apache/hadoop/minikdc/MiniKdc.classMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xmlMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
However when I run the "Yarn" integration test ("mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests"), I am getting compilation errors. I have found that the runtime classpath does not include the "flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar" from "flink-shaded-include-yarn-tests/target" location. The classpath points to local repository jar (/home/vijay/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar) which does not have the "minikdc" classes.
Not sure how the classpath is derived but clearly something is missing. You could try by just importing "org.apache.hadoop.minikdc.MiniKdc" in one of the Yarn integration test class file and run "mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests" command
cd ~/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT
$ jar tf flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdc
RegardsVijay
On Tuesday, June 21, 2016 9:07 AM, Robert Metzger <rm...@apache.org> wrote:
Hi,
This this is how I did it:
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
pwd
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
flink-shaded-include-yarn-tests ---
[INFO] Deleting
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target
[INFO]
[INFO] --- maven-checkstyle-plugin:2.16:check (validate) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
flink-shaded-include-yarn-tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
flink-shaded-include-yarn-tests ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources)
@ flink-shaded-include-yarn-tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
flink-shaded-include-yarn-tests ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @
flink-shaded-include-yarn-tests ---
[INFO] Building jar:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (shade-hadoop) @
flink-shaded-include-yarn-tests ---
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.3.0 in the
shaded jar.
[INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.5 from the shaded
jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the
shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.1 from the
shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded
jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-json:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.8.3 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.8.3 from the shaded
jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Including asm:asm:jar:3.1 in the shaded jar.
[INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar.
[INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded
jar.
[INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the
shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded
jar.
[INFO] Excluding com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the
shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.7 from
the shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the
shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.8.3 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-core-asl:jar:1.8.8 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 from the
shaded jar.
[INFO] Excluding org.apache.avro:avro:jar:1.7.6 from the shaded jar.
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.3 from the
shaded jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.0.5 from the shaded
jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:2.3.0 in the shaded jar.
[INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded
jar.
[INFO] Including io.netty:netty:jar:3.7.0.Final in the shaded jar.
[INFO] Excluding org.apache.commons:commons-compress:jar:1.4.1 from the
shaded jar.
[INFO] Excluding org.tukaani:xz:jar:1.0 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:2.3.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.3.0 in the shaded
jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-tests:test-jar:tests:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-minicluster:jar:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:test-jar:tests:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.3.0 in the
shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded
jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar:tests:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.3.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.3.0 in the
shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.3.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0
in the shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-hs:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-minikdc:jar:2.3.0 in the shaded
jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from the shaded
jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-constants:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-i18n:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-client-api:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20 from the shaded
jar.
[INFO] Excluding org.apache.directory.api:api-ldap-model:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.mina:mina-core:jar:2.0.7 from the shaded jar.
[INFO] Excluding net.sf.ehcache:ehcache-core:jar:2.4.4 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptor-kerberos:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-core:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-authn:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-authz:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-changelog:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-collective:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-event:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-exception:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-journal:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-normalization:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-operational:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-referral:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-schema:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-subtree:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-trigger:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-trigger:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-shared:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-kerberos:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-shared:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-ldif-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-xdbm-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-mavibot-partition:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding org.apache.directory.mavibot:mavibot:jar:1.0.0-M1 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-avl:jar:2.0.0-M15 from the shaded
jar.
[INFO] Excluding org.apache.directory.api:api-all:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding commons-pool:commons-pool:jar:1.6 from the shaded jar.
[INFO] Excluding antlr:antlr:jar:2.7.7 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-jdbm-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-ldap:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-codec-api:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-codec:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-extras-sp:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding bouncycastle:bcprov-jdk15:jar:140 from the shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.3.0 in the
shaded jar.
[INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from
the shaded jar.
[INFO] Excluding com.google.inject:guice:jar:3.0 from the shaded jar.
[INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar.
[INFO] Excluding aopalliance:aopalliance:jar:1.0 from the shaded jar.
[INFO] Excluding com.sun.jersey.contribs:jersey-guice:jar:1.9 from the
shaded jar.
[INFO] Excluding javax.xml.bind:jaxb-api:jar:2.2.2 from the shaded jar.
[INFO] Excluding javax.xml.stream:stax-api:jar:1.0-2 from the shaded jar.
[INFO] Excluding javax.activation:activation:jar:1.1 from the shaded jar.
[INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.3.0
in the shaded jar.
[INFO] Excluding org.apache.flink:force-shading:jar:1.1-SNAPSHOT from the
shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded
jar.
[INFO] Excluding org.apache.commons:commons-lang3:jar:3.3.2 from the shaded
jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[WARNING] hadoop-common-2.3.0-tests.jar, hadoop-common-2.3.0.jar define 37
overlapping classes:
[WARNING] - org.apache.hadoop.io.serializer.avro.AvroRecord
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpcProto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$Builder
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$1
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EchoRequestProto$Builder
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProtoOrBuilder
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos
[WARNING] - 27 more...
[WARNING] hadoop-yarn-common-2.3.0.jar, hadoop-yarn-api-2.3.0.jar define 3
overlapping classes:
[WARNING] - org.apache.hadoop.yarn.util.package-info
[WARNING] - org.apache.hadoop.yarn.factories.package-info
[WARNING] - org.apache.hadoop.yarn.factory.providers.package-info
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
with
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-javadoc-plugin:2.9.1:jar (attach-javadocs) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (integration-tests) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- japicmp-maven-plugin:0.7.0:cmp (default) @
flink-shaded-include-yarn-tests ---
[INFO] Skipping execution because parameter 'skip' was set to true.
[INFO]
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @
flink-shaded-include-yarn-tests ---
[INFO] Installing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
to
/home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
[INFO] Installing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
to
/home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.pom
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 14.906 s
[INFO] Finished at: 2016-06-21T18:06:43+02:00
[INFO] Final Memory: 33M/420M
[INFO]
------------------------------------------------------------------------
mvn clean install 55.91s user 1.63s system 354% cpu 16.229 total
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar| grep minikdc
minikdc-krb5.conf
minikdc.ldiff
org/apache/hadoop/minikdc/
org/apache/hadoop/minikdc/KerberosSecurityTestcase.class
org/apache/hadoop/minikdc/MiniKdc$1.class
org/apache/hadoop/minikdc/MiniKdc.class
META-INF/maven/org.apache.hadoop/hadoop-minikdc/
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
On Tue, Jun 21, 2016 at 5:47 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> Thanks Rob for trying. In my machine, I don't see any of minikdc related
> files in the target jar.
>
> How did you run the build?
>
> Regards
> Vijay
>
>
> On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Cool, thank you.
>
> After adding the line you've mentioned, my jar contains the missing
> MiniKdc class:
>
> robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master]
> % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep
> minikdc
> minikdc-krb5.conf
> minikdc.ldiff
> org/apache/hadoop/minikdc/
> org/apache/hadoop/minikdc/KerberosSecurityTestcase.class
> org/apache/hadoop/minikdc/MiniKdc$1.class
> org/apache/hadoop/minikdc/MiniKdc.class
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
> robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master]
> %
>
> Is the same true on your machine?
>
>
>
>
> On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com.invalid> wrote:
>
> Hi Rob,
> You need to include below lines to the pom.xml to resolve the chain
> dependency error.
> <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC
> dependency error --> <build> <plugins> <plugin>
> <groupId>org.apache.felix</groupId>
> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited>
> <extensions>true</extensions> </plugin> </plugins> </build>
> RegardsVijay
>
> On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi,
> I didn't even get the shading module to build:
> [INFO]
> ------------------------------------------------------------------------[INFO]
> Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO]
> ------------------------------------------------------------------------[INFO]
> ------------------------------------------------------------------------[INFO]
> BUILD FAILURE[INFO]
> ------------------------------------------------------------------------[INFO]
> Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO]
> Final Memory: 16M/303M[INFO]
> ------------------------------------------------------------------------[ERROR]
> Failed to execute goal on project flink-shaded-include-yarn-tests: Could
> not resolve dependencies for project
> org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure
> to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1][ERROR]
>
> How did you do it?
> Regards,Robert
>
> On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hi Rob,
> Yes I checked but the result jar does not contain the classes from
> hadoop-minikdc package.
> RegardsVijay
>
> On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi Vijay,
> did you check if the artifact produced by the
> "flink-shaded-include-yarn-tests" module contains the required classes?
> Regards,Robert
>
> On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hello,
> I was trying to include "hadoop-minikdc" component to Yarn test framework
> by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
>
> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
> The dependency inclusion seems to be working from IDE. IntelliJ picked up
> the jar and did not report any compilation issue when I create a new test
> program in "flink-yarn-tests" module and try to use the minikdc class.
> However, when I try to run the test using maven command, I am getting
> compilation error for the MiniKdc package.
> Could someone please share some insights on how the dependency inclusion
> works with the shaded hadoop components?
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
> -Dtest=YARNSessionFIFOITCase
>
> [ERROR] COMPILATION ERROR : [INFO]
> -------------------------------------------------------------[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
> package org.apache.hadoop.minikdc does not exist[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
> cannot find symbol symbol: variable MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
> RegardsVijay
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Shading Plugin + Hadoop Dependency
Posted by Robert Metzger <rm...@apache.org>.
Hi,
This this is how I did it:
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
pwd
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
flink-shaded-include-yarn-tests ---
[INFO] Deleting
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target
[INFO]
[INFO] --- maven-checkstyle-plugin:2.16:check (validate) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
flink-shaded-include-yarn-tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
flink-shaded-include-yarn-tests ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources)
@ flink-shaded-include-yarn-tests ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
flink-shaded-include-yarn-tests ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @
flink-shaded-include-yarn-tests ---
[INFO] Building jar:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (shade-hadoop) @
flink-shaded-include-yarn-tests ---
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.3.0 in the
shaded jar.
[INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.5 from the shaded
jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the
shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.1 from the
shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded
jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-json:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.8.3 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.8.3 from the shaded
jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Including asm:asm:jar:3.1 in the shaded jar.
[INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar.
[INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded
jar.
[INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the
shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded
jar.
[INFO] Excluding com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the
shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.7 from
the shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the
shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.8.3 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-core-asl:jar:1.8.8 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 from the
shaded jar.
[INFO] Excluding org.apache.avro:avro:jar:1.7.6 from the shaded jar.
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.3 from the
shaded jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.0.5 from the shaded
jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:2.3.0 in the shaded jar.
[INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded
jar.
[INFO] Including io.netty:netty:jar:3.7.0.Final in the shaded jar.
[INFO] Excluding org.apache.commons:commons-compress:jar:1.4.1 from the
shaded jar.
[INFO] Excluding org.tukaani:xz:jar:1.0 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:2.3.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.3.0 in the shaded
jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-tests:test-jar:tests:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-minicluster:jar:2.3.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:test-jar:tests:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.3.0 in the
shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded
jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar:tests:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.3.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.3.0 in the
shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.3.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0
in the shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-hs:jar:2.3.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-minikdc:jar:2.3.0 in the shaded
jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from the shaded
jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-constants:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-i18n:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-client-api:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20 from the shaded
jar.
[INFO] Excluding org.apache.directory.api:api-ldap-model:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.mina:mina-core:jar:2.0.7 from the shaded jar.
[INFO] Excluding net.sf.ehcache:ehcache-core:jar:2.4.4 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptor-kerberos:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-core:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-authn:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-authz:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-changelog:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-collective:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-event:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-exception:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-journal:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-normalization:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-operational:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-referral:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-schema:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-subtree:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-interceptors-trigger:jar:2.0.0-M15
from the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-trigger:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-shared:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-kerberos:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-shared:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-ldif-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-xdbm-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-mavibot-partition:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding org.apache.directory.mavibot:mavibot:jar:1.0.0-M1 from the
shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-core-avl:jar:2.0.0-M15 from the shaded
jar.
[INFO] Excluding org.apache.directory.api:api-all:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding commons-pool:commons-pool:jar:1.6 from the shaded jar.
[INFO] Excluding antlr:antlr:jar:2.7.7 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-jdbm-partition:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2
from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-protocol-ldap:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20 from
the shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-codec-api:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding
org.apache.directory.api:api-ldap-extras-codec:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-ldap-extras-sp:jar:1.0.0-M20
from the shaded jar.
[INFO] Excluding bouncycastle:bcprov-jdk15:jar:140 from the shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.3.0 in the
shaded jar.
[INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from
the shaded jar.
[INFO] Excluding com.google.inject:guice:jar:3.0 from the shaded jar.
[INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar.
[INFO] Excluding aopalliance:aopalliance:jar:1.0 from the shaded jar.
[INFO] Excluding com.sun.jersey.contribs:jersey-guice:jar:1.9 from the
shaded jar.
[INFO] Excluding javax.xml.bind:jaxb-api:jar:2.2.2 from the shaded jar.
[INFO] Excluding javax.xml.stream:stax-api:jar:1.0-2 from the shaded jar.
[INFO] Excluding javax.activation:activation:jar:1.1 from the shaded jar.
[INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.3.0
in the shaded jar.
[INFO] Excluding org.apache.flink:force-shading:jar:1.1-SNAPSHOT from the
shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded
jar.
[INFO] Excluding org.apache.commons:commons-lang3:jar:3.3.2 from the shaded
jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[WARNING] hadoop-common-2.3.0-tests.jar, hadoop-common-2.3.0.jar define 37
overlapping classes:
[WARNING] - org.apache.hadoop.io.serializer.avro.AvroRecord
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpcProto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$Builder
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$1
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto$1
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EchoRequestProto$Builder
[WARNING] -
org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProtoOrBuilder
[WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos
[WARNING] - 27 more...
[WARNING] hadoop-yarn-common-2.3.0.jar, hadoop-yarn-api-2.3.0.jar define 3
overlapping classes:
[WARNING] - org.apache.hadoop.yarn.util.package-info
[WARNING] - org.apache.hadoop.yarn.factories.package-info
[WARNING] - org.apache.hadoop.yarn.factory.providers.package-info
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
with
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-javadoc-plugin:2.9.1:jar (attach-javadocs) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (integration-tests) @
flink-shaded-include-yarn-tests ---
[INFO]
[INFO] --- japicmp-maven-plugin:0.7.0:cmp (default) @
flink-shaded-include-yarn-tests ---
[INFO] Skipping execution because parameter 'skip' was set to true.
[INFO]
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @
flink-shaded-include-yarn-tests ---
[INFO] Installing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
to
/home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
[INFO] Installing
/home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml
to
/home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.pom
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 14.906 s
[INFO] Finished at: 2016-06-21T18:06:43+02:00
[INFO] Final Memory: 33M/420M
[INFO]
------------------------------------------------------------------------
mvn clean install 55.91s user 1.63s system 354% cpu 16.229 total
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar| grep minikdc
minikdc-krb5.conf
minikdc.ldiff
org/apache/hadoop/minikdc/
org/apache/hadoop/minikdc/KerberosSecurityTestcase.class
org/apache/hadoop/minikdc/MiniKdc$1.class
org/apache/hadoop/minikdc/MiniKdc.class
META-INF/maven/org.apache.hadoop/hadoop-minikdc/
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
On Tue, Jun 21, 2016 at 5:47 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com> wrote:
> Thanks Rob for trying. In my machine, I don't see any of minikdc related
> files in the target jar.
>
> How did you run the build?
>
> Regards
> Vijay
>
>
> On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Cool, thank you.
>
> After adding the line you've mentioned, my jar contains the missing
> MiniKdc class:
>
> robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master]
> % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep
> minikdc
> minikdc-krb5.conf
> minikdc.ldiff
> org/apache/hadoop/minikdc/
> org/apache/hadoop/minikdc/KerberosSecurityTestcase.class
> org/apache/hadoop/minikdc/MiniKdc$1.class
> org/apache/hadoop/minikdc/MiniKdc.class
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
> robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master]
> %
>
> Is the same true on your machine?
>
>
>
>
> On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com.invalid> wrote:
>
> Hi Rob,
> You need to include below lines to the pom.xml to resolve the chain
> dependency error.
> <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC
> dependency error --> <build> <plugins> <plugin>
> <groupId>org.apache.felix</groupId>
> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited>
> <extensions>true</extensions> </plugin> </plugins> </build>
> RegardsVijay
>
> On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi,
> I didn't even get the shading module to build:
> [INFO]
> ------------------------------------------------------------------------[INFO]
> Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO]
> ------------------------------------------------------------------------[INFO]
> ------------------------------------------------------------------------[INFO]
> BUILD FAILURE[INFO]
> ------------------------------------------------------------------------[INFO]
> Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO]
> Final Memory: 16M/303M[INFO]
> ------------------------------------------------------------------------[ERROR]
> Failed to execute goal on project flink-shaded-include-yarn-tests: Could
> not resolve dependencies for project
> org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure
> to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1][ERROR]
>
> How did you do it?
> Regards,Robert
>
> On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hi Rob,
> Yes I checked but the result jar does not contain the classes from
> hadoop-minikdc package.
> RegardsVijay
>
> On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi Vijay,
> did you check if the artifact produced by the
> "flink-shaded-include-yarn-tests" module contains the required classes?
> Regards,Robert
>
> On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hello,
> I was trying to include "hadoop-minikdc" component to Yarn test framework
> by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
>
> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
> The dependency inclusion seems to be working from IDE. IntelliJ picked up
> the jar and did not report any compilation issue when I create a new test
> program in "flink-yarn-tests" module and try to use the minikdc class.
> However, when I try to run the test using maven command, I am getting
> compilation error for the MiniKdc package.
> Could someone please share some insights on how the dependency inclusion
> works with the shaded hadoop components?
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
> -Dtest=YARNSessionFIFOITCase
>
> [ERROR] COMPILATION ERROR : [INFO]
> -------------------------------------------------------------[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
> package org.apache.hadoop.minikdc does not exist[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
> cannot find symbol symbol: variable MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
> RegardsVijay
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Shading Plugin + Hadoop Dependency
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Thanks Rob for trying. In my machine, I don't see any of minikdc related files in the target jar.
How did you run the build?
RegardsVijay
On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <rm...@apache.org> wrote:
Cool, thank you.
After adding the line you've mentioned, my jar contains the missing MiniKdc class:
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdcminikdc-krb5.confminikdc.ldifforg/apache/hadoop/minikdc/org/apache/hadoop/minikdc/KerberosSecurityTestcase.classorg/apache/hadoop/minikdc/MiniKdc$1.classorg/apache/hadoop/minikdc/MiniKdc.classMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xmlMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.propertiesrobert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
Is the same true on your machine?
On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hi Rob,
You need to include below lines to the pom.xml to resolve the chain dependency error.
<!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC dependency error --> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> <extensions>true</extensions> </plugin> </plugins> </build>
RegardsVijay
On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rm...@apache.org> wrote:
Hi,
I didn't even get the shading module to build:
[INFO] ------------------------------------------------------------------------[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] Final Memory: 16M/303M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1][ERROR]
How did you do it?
Regards,Robert
On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hi Rob,
Yes I checked but the result jar does not contain the classes from hadoop-minikdc package.
RegardsVijay
On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org> wrote:
Hi Vijay,
did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes?
Regards,Robert
On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class.
However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package.
Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components?
mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase
[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
RegardsVijay
Re: Shading Plugin + Hadoop Dependency
Posted by Robert Metzger <rm...@apache.org>.
Cool, thank you.
After adding the line you've mentioned, my jar contains the missing MiniKdc
class:
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep
minikdc
minikdc-krb5.conf
minikdc.ldiff
org/apache/hadoop/minikdc/
org/apache/hadoop/minikdc/KerberosSecurityTestcase.class
org/apache/hadoop/minikdc/MiniKdc$1.class
org/apache/hadoop/minikdc/MiniKdc.class
META-INF/maven/org.apache.hadoop/hadoop-minikdc/
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml
META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] %
Is the same true on your machine?
On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com.invalid> wrote:
> Hi Rob,
> You need to include below lines to the pom.xml to resolve the chain
> dependency error.
> <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC
> dependency error --> <build> <plugins> <plugin>
> <groupId>org.apache.felix</groupId>
> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited>
> <extensions>true</extensions> </plugin> </plugins> </build>
> RegardsVijay
>
> On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi,
> I didn't even get the shading module to build:
> [INFO]
> ------------------------------------------------------------------------[INFO]
> Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO]
> ------------------------------------------------------------------------[INFO]
> ------------------------------------------------------------------------[INFO]
> BUILD FAILURE[INFO]
> ------------------------------------------------------------------------[INFO]
> Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO]
> Final Memory: 16M/303M[INFO]
> ------------------------------------------------------------------------[ERROR]
> Failed to execute goal on project flink-shaded-include-yarn-tests: Could
> not resolve dependencies for project
> org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure
> to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1][ERROR]
>
> How did you do it?
> Regards,Robert
>
> On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hi Rob,
> Yes I checked but the result jar does not contain the classes from
> hadoop-minikdc package.
> RegardsVijay
>
> On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi Vijay,
> did you check if the artifact produced by the
> "flink-shaded-include-yarn-tests" module contains the required classes?
> Regards,Robert
>
> On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hello,
> I was trying to include "hadoop-minikdc" component to Yarn test framework
> by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
>
> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
> The dependency inclusion seems to be working from IDE. IntelliJ picked up
> the jar and did not report any compilation issue when I create a new test
> program in "flink-yarn-tests" module and try to use the minikdc class.
> However, when I try to run the test using maven command, I am getting
> compilation error for the MiniKdc package.
> Could someone please share some insights on how the dependency inclusion
> works with the shaded hadoop components?
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
> -Dtest=YARNSessionFIFOITCase
>
> [ERROR] COMPILATION ERROR : [INFO]
> -------------------------------------------------------------[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
> package org.apache.hadoop.minikdc does not exist[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
> cannot find symbol symbol: variable MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
> RegardsVijay
>
>
>
>
>
>
>
>
>
Re: Shading Plugin + Hadoop Dependency
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Hi Rob,
You need to include below lines to the pom.xml to resolve the chain dependency error.
<!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC dependency error --> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> <extensions>true</extensions> </plugin> </plugins> </build>
RegardsVijay
On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rm...@apache.org> wrote:
Hi,
I didn't even get the shading module to build:
[INFO] ------------------------------------------------------------------------[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] Final Memory: 16M/303M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1][ERROR]
How did you do it?
Regards,Robert
On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hi Rob,
Yes I checked but the result jar does not contain the classes from hadoop-minikdc package.
RegardsVijay
On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org> wrote:
Hi Vijay,
did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes?
Regards,Robert
On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class.
However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package.
Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components?
mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase
[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
RegardsVijay
Re: Shading Plugin + Hadoop Dependency
Posted by Robert Metzger <rm...@apache.org>.
Hi,
I didn't even get the shading module to build:
[INFO]
------------------------------------------------------------------------
[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 0.756 s
[INFO] Finished at: 2016-06-21T17:25:00+02:00
[INFO] Final Memory: 16M/303M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests:
Could not resolve dependencies for project
org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure
to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in
https://repo.maven.apache.org/maven2 was cached in the local repository,
resolution will not be reattempted until the update interval of central has
elapsed or updates are forced -> [Help 1]
[ERROR]
How did you do it?
Regards,
Robert
On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com.invalid> wrote:
> Hi Rob,
> Yes I checked but the result jar does not contain the classes from
> hadoop-minikdc package.
> RegardsVijay
>
> On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>
> Hi Vijay,
> did you check if the artifact produced by the
> "flink-shaded-include-yarn-tests" module contains the required classes?
> Regards,Robert
>
> On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan
> <vi...@yahoo.com.invalid> wrote:
>
> Hello,
> I was trying to include "hadoop-minikdc" component to Yarn test framework
> by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
>
> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
> The dependency inclusion seems to be working from IDE. IntelliJ picked up
> the jar and did not report any compilation issue when I create a new test
> program in "flink-yarn-tests" module and try to use the minikdc class.
> However, when I try to run the test using maven command, I am getting
> compilation error for the MiniKdc package.
> Could someone please share some insights on how the dependency inclusion
> works with the shaded hadoop components?
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
> -Dtest=YARNSessionFIFOITCase
>
> [ERROR] COMPILATION ERROR : [INFO]
> -------------------------------------------------------------[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
> package org.apache.hadoop.minikdc does not exist[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
> cannot find symbol symbol: variable MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
> RegardsVijay
>
>
>
>
>
Re: Shading Plugin + Hadoop Dependency
Posted by Vijay Srinivasaraghavan <vi...@yahoo.com.INVALID>.
Hi Rob,
Yes I checked but the result jar does not contain the classes from hadoop-minikdc package.
RegardsVijay
On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rm...@apache.org> wrote:
Hi Vijay,
did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes?
Regards,Robert
On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <vi...@yahoo.com.invalid> wrote:
Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class.
However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package.
Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components?
mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase
[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
RegardsVijay
Re: Shading Plugin + Hadoop Dependency
Posted by Robert Metzger <rm...@apache.org>.
Hi Vijay,
did you check if the artifact produced by the
"flink-shaded-include-yarn-tests" module contains the required classes?
Regards,
Robert
On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com.invalid> wrote:
> Hello,
> I was trying to include "hadoop-minikdc" component to Yarn test framework
> by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file.
>
> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency>
> The dependency inclusion seems to be working from IDE. IntelliJ picked up
> the jar and did not report any compilation issue when I create a new test
> program in "flink-yarn-tests" module and try to use the minikdc class.
> However, when I try to run the test using maven command, I am getting
> compilation error for the MiniKdc package.
> Could someone please share some insights on how the dependency inclusion
> works with the shaded hadoop components?
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
> -Dtest=YARNSessionFIFOITCase
>
> [ERROR] COMPILATION ERROR : [INFO]
> -------------------------------------------------------------[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
> package org.apache.hadoop.minikdc does not exist[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
> cannot find symbol symbol: variable MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
> /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
> cannot find symbol symbol: class MiniKdc location: class
> org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
> RegardsVijay