You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by tedxia <gi...@git.apache.org> on 2015/06/19 10:22:04 UTC
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
GitHub user tedxia opened a pull request:
https://github.com/apache/storm/pull/600
STORM-903: fix java.lang.ClassNotFoundException with org.apache.commo…
…ns.codec.binary.Base64 when set ui.filter to org.apache.hadoop.security.authentication.server.AuthenticationFilter
PR for [STORM-903](https://issues.apache.org/jira/browse/STORM-903)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tedxia/storm ted-fix-missing-codex-when-use-ui-filter-as-AuthenticationFilter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/600.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #600
----
commit 155478204705e78777ccb5972ac495eb28f700d6
Author: xiajun <xi...@xiaomi.com>
Date: 2015-06-19T08:18:26Z
STORM-903: fix java.lang.ClassNotFoundException with org.apache.commons.codec.binary.Base64 when set ui.filter to org.apache.hadoop.security.authentication.server.AuthenticationFilter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115940765
@revans2 agree on not shading. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115349321
@revans2 not sure this addresses the issue for using KerberosFilter from hadoop for securing UI when used in kerberos cluster. We excluded commons-codec from hadoop-auth because storm is including it in the classpath https://github.com/apache/storm/blob/master/storm-core/pom.xml#L259
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115699285
@tedxia I don't really like the idea. I think we should make it simple for people to do that, but the problem is hdfs and hbase are more than just a set of jars. The configuration required to make them work is often extensive.
Perhaps what we can do is to have storm-hdfs and storm-hbase in extlib-daemon by default, and inside the default storm-env.sh we look for hadoop and hbase on the PATH, and if they are found we append the result of `hadoop classpath` and `hbase classpath` to STORM_EXT_CLASSPATH_DAEMON.
That way they are there to be used if hadoop and hbase are setup on the node and we are not tied to a very specific version or configuration of hadoop and hbase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #600: STORM-903: fix java.lang.ClassNotFoundException with org.a...
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the issue:
https://github.com/apache/storm/pull/600
I have closed this PR due to inactivity. Jira STORM-903 remains open. Please re-open or create a new PR if we want to fix this.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115355904
@harshach If you are pulling in an external library, like the KerberosFilter, you are responsible for making sure all of its dependencies are on the classpath. If we have shaded some of them, then you need to pull them over and include them. If it is something that a worker does not need access to, like the KerberosFilter, then you can include it in the extlib-damone directory where it will not show up on the worker's classpath.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115360459
@revans2 we are shipping security in 0.10 that means we need to provide a way for UI daemon to be secured hence the reason we are including kerberos filter as part of storm-core/pom.xml that its easier for users to just add a config ui.filter and use kerberos filter. Now the patch STORM-848 which shaded and removed commons-codec should also remove the exclusion of commons-codec from hadoop-auth. I don't think making this as user's responsibility makes sense here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115845268
@harshach which exclude are you talking about? If you want to make a change quickly I am OK with that. I am just not totally sure what the impact that this is going to have. Will we now have commons-codec as a dependency that we didn't before? Will we now have a version of hadoop-auth with classes rewritten to point to the shaded commons-codec?
If it is the former, then we will be stuck with commons-codec being a dependency until we go to another major version. If it is the former I am a bit nervous about how hadoop might react to it, but not very nervous.
I am fine either way, I just want to know what the impact is, and then have someone put together a pull request we can review it and check it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115866496
@revans2 this line https://github.com/apache/storm/blob/master/storm-core/pom.xml#L259 so the reason we excluded is there is commons-codec in path from storm as its dependency before we shaded it. Now since we removed it hadoop-auth is throwing this error that its unable to find.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115875141
So this pull request is what you want to have happen. I am OK with that, but if we are going to have commons-codec on the classpath, then lets not bother shading it at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #600: STORM-903: fix java.lang.ClassNotFoundException wit...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/600
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115872534
@harshach so what you are saying is that if we should remove the exclude, but leave in the shading. This means we will have two copies of commons-codec, one shaded that storm will use, and one that hadoop-auth will use that is not shaded and will be on the path?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by tedxia <gi...@git.apache.org>.
Github user tedxia commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115479671
@revans2 what do you think about add storm-hdfs and storm-hbase and all of their dependencies to extlib-daemon by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115263584
@tedxia We shaded this for a reason. If AutoHDFS is not able to find it, then we should look at updating AutoHDFS to use the shaded version of the library. I am not sure if that is possible or not. It depends on if it is us that is using commons-codec directly or if it is another dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by tedxia <gi...@git.apache.org>.
Github user tedxia commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115479231
@revans2 thanks for your replay. But I still think we must add storm-hdfs and all of its dependencies to lib or extlib, and storm user must add storm-hdfs as topology's dependency as it use HdfsBolt and so on. The reason why we should add storm-hdfs to daemon's CLASS_PATH is that daemons such as nimbus and supervisor will directly call AutoHDFS‘s interface. I think it is a good idea if we add storm-hdfs and storm-hbase and all of their dependencies to extlib-daemon by default. The jars that have been shaded by storm-core must also add to extlib-daemon, if they are depended by storm-hdfs or storm-hbase, just because AutoHDFS and AutoHBase load by class loader and they can't identify the shaded jars.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115269037
@tedxia sorry I am wrong. hadoop-hdfs itself depends on commons-codec. But the issue is around the packaging of auto-hdfs.
For most users who want to use storm-hdfs in their topologies they will include it as a dependency and include it as part of an uber-jar. This will pull in commons-codec correctly. The issue is if you want to just include it in the classpath by default. If all you do is take storm-hdfs and put it in the lib directory you will get these errors. If you don't want your users to have to package auto-hdfs with their topology then you need to include storm-hdfs, and all of its dependencies in the lib directory or set STORM_EXT_CLASSPATH before launching the various daemons. This is nice because it lets you potentially query a hadoop install already on the box for it's classpath instead of trying to figure it all out yourself.
Alternatively you could just set it up for storm-hdfs to be on the daemons classpath and not pollute the worker classpath. You can do this by using STORM_EXT_CLASSPATH_DAEMON and/or extlib-daemon directory. What we really need is some better documentation about how to install and setup atuo-hdfs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-114657493
@kishorvpatil I think you may want to take care of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #600: STORM-903: fix java.lang.ClassNotFoundException with org.a...
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the issue:
https://github.com/apache/storm/pull/600
@revans2, @harshach , @tedxia
This pull request is stale, and it looks as if we are pulling in commons-codec [explicitly](https://github.com/apache/storm/blob/0fdad2c0ff357c431454128dd937edf2876a5fb5/storm-core/pom.xml#L330-L333) now:
```
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.storm:storm-core:jar:2.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.zookeeper:zookeeper:jar -> duplicate declaration of version (?) @ org.apache.storm:storm-core:[unknown-version], /Users/derekd/devel/storm/storm-core/pom.xml, line 282, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] --------------------< org.apache.storm:storm-core >---------------------
[INFO] Building Storm Core 2.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ storm-core ---
[INFO] org.apache.storm:storm-core:jar:2.0.0-SNAPSHOT
[INFO] +- org.apache.storm:storm-clojure:jar:2.0.0-SNAPSHOT:compile
[INFO] +- org.apache.storm:storm-server:jar:2.0.0-SNAPSHOT:compile
[INFO] | +- org.apache.storm:storm-client:jar:2.0.0-SNAPSHOT:compile
[INFO] | | +- org.jctools:jctools-core:jar:2.0.1:compile
[INFO] | | \- javax.xml.bind:jaxb-api:jar:2.3.0:compile
[INFO] | \- org.rocksdb:rocksdbjni:jar:5.8.6:compile
[INFO] +- uk.org.lidalia:sysout-over-slf4j:jar:1.0.2:compile
[INFO] +- log4j:log4j:jar:1.2.17:test
[INFO] +- org.clojure:clojure:jar:1.7.0:compile
[INFO] +- clj-time:clj-time:jar:0.8.0:compile
[INFO] +- compojure:compojure:jar:1.1.9:compile
[INFO] | +- org.clojure:tools.macro:jar:0.1.0:compile
[INFO] | \- clout:clout:jar:1.2.0:compile
[INFO] +- hiccup:hiccup:jar:0.3.6:compile
[INFO] +- ring:ring-core:jar:1.6.2:compile
[INFO] | +- ring:ring-codec:jar:1.0.1:compile
[INFO] | +- crypto-random:crypto-random:jar:1.2.0:compile
[INFO] | \- crypto-equality:crypto-equality:jar:1.0.0:compile
[INFO] +- ring:ring-devel:jar:1.6.2:compile
[INFO] | +- clj-stacktrace:clj-stacktrace:jar:0.2.8:compile
[INFO] | \- ns-tracker:ns-tracker:jar:0.3.1:compile
[INFO] | +- org.clojure:tools.namespace:jar:0.2.11:compile
[INFO] | \- org.clojure:java.classpath:jar:0.2.3:compile
[INFO] +- ring:ring-jetty-adapter:jar:1.6.2:compile
[INFO] | \- ring:ring-servlet:jar:1.6.2:compile
[INFO] +- ring:ring-json:jar:0.4.0:compile
[INFO] | \- cheshire:cheshire:jar:5.5.0:compile
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.9.4:compile
[INFO] | \- tigris:tigris:jar:0.1.1:compile
[INFO] +- org.clojure:tools.logging:jar:0.2.3:compile
[INFO] +- org.clojure:math.numeric-tower:jar:0.0.1:compile
[INFO] +- org.clojure:tools.cli:jar:0.2.4:compile
[INFO] +- org.clojure:tools.nrepl:jar:0.2.3:test
[INFO] +- clojure-complete:clojure-complete:jar:0.2.3:test
[INFO] +- org.clojars.runa:conjure:jar:2.1.3:test
[INFO] +- org.clojure:clojure-contrib:jar:1.2.0:test
[INFO] +- org.clojure:data.codec:jar:0.1.0:test
[INFO] +- org.clojure:java.jmx:jar:0.3.1:compile
[INFO] +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] +- commons-io:commons-io:jar:2.5:compile
[INFO] +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.commons:commons-exec:jar:1.1:compile
[INFO] +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] +- commons-lang:commons-lang:jar:2.5:compile
[INFO] +- org.apache.thrift:libthrift:jar:0.9.3:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.3.3:compile
[INFO] | | \- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | \- org.apache.httpcomponents:httpcore:jar:4.4.1:compile
[INFO] +- org.apache.curator:curator-framework:jar:4.0.1:compile
[INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | \- jline:jline:jar:0.9.94:compile
[INFO] +- org.apache.curator:curator-recipes:jar:4.0.1:compile
[INFO] +- org.apache.curator:curator-test:jar:2.12.0:test
[INFO] | \- org.javassist:javassist:jar:3.18.1-GA:test
[INFO] +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] +- com.twitter:carbonite:jar:1.5.0:compile
[INFO] | +- com.esotericsoftware:kryo:jar:3.0.3:compile
[INFO] | | +- com.esotericsoftware:reflectasm:jar:1.10.1:compile
[INFO] | | | \- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] | | \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] | \- com.twitter:chill-java:jar:0.8.0:compile
[INFO] | \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] +- org.yaml:snakeyaml:jar:1.11:compile
[INFO] +- com.lmax:disruptor:jar:3.3.2:compile
[INFO] +- org.jgrapht:jgrapht-core:jar:0.9.0:compile
[INFO] +- com.google.guava:guava:jar:16.0.1:compile
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.8.2:compile
[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.8.2:compile
[INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.8.2:compile
[INFO] +- org.slf4j:log4j-over-slf4j:jar:1.6.6:compile
[INFO] +- io.netty:netty:jar:3.9.9.Final:compile
[INFO] +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile
[INFO] +- metrics-clojure:metrics-clojure:jar:2.5.1:compile
[INFO] +- junit:junit:jar:4.11:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.mockito:mockito-core:jar:2.10.0:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.7.4:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.7.4:test
[INFO] | \- org.objenesis:objenesis:jar:2.6:compile
[INFO] +- org.hamcrest:java-hamcrest:jar:2.0.0.0:test
[INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] +- joda-time:joda-time:jar:2.3:compile
[INFO] +- org.eclipse.jetty:jetty-server:jar:9.4.7.v20170914:compile
[INFO] | +- org.eclipse.jetty:jetty-http:jar:9.4.7.v20170914:compile
[INFO] | \- org.eclipse.jetty:jetty-io:jar:9.4.7.v20170914:compile
[INFO] +- org.eclipse.jetty:jetty-util:jar:9.4.7.v20170914:compile
[INFO] +- org.eclipse.jetty:jetty-servlet:jar:9.4.7.v20170914:compile
[INFO] | \- org.eclipse.jetty:jetty-security:jar:9.4.7.v20170914:compile
[INFO] +- org.eclipse.jetty:jetty-servlets:jar:9.4.7.v20170914:compile
[INFO] | \- org.eclipse.jetty:jetty-continuation:jar:9.4.7.v20170914:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.9.4:compile
[INFO] +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.9.4:compile
[INFO] +- org.apache.curator:curator-client:jar:4.0.1:compile
[INFO] +- commons-fileupload:commons-fileupload:jar:1.3.2:compile
[INFO] +- commons-codec:commons-codec:jar:1.6:compile
[INFO] \- org.apache.hadoop:hadoop-auth:jar:2.6.1:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.071 s
[INFO] Finished at: 2018-03-22T09:33:16-05:00
[INFO] ------------------------------------------------------------------------
```
Should this pull request be closed?
---
[GitHub] storm pull request: STORM-903: fix java.lang.ClassNotFoundExceptio...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/600#issuecomment-115835563
@revans2 what about the hadoop-auth that we are including right now can we atleast remove the "exclude commons-codec" from hadoop-auth dependency. It will give users a filter to use it with UI in secure mode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---