You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/08/26 09:50:52 UTC
[GitHub] [incubator-druid] Dylan1312 edited a comment on issue #8339: Don't
inadvertently exclude hadoop-client-core dependency
Dylan1312 edited a comment on issue #8339: Don't inadvertently exclude hadoop-client-core dependency
URL: https://github.com/apache/incubator-druid/pull/8339#issuecomment-524797775
Hey @clintropolis, no problem at all and thanks for looking! Happy to bump the milestone on this since we haven't heard of anyone else that's affected.
We're using a Cloudera distro of 2.6. I can get a full example of an indexing spec but I think the most pertinent thing that's potentially different is setting mapreduce.job.classloader = true (to avoid a jackson dependency conflict). Guessing that without classloader isolation the missing classes would be present in the hadoop install itself.
Example below showing the difference between a build of this branch and our latest release, note hadoop-mapreduce-client-core-2.8.3.jar isn't present within druid-hdfs-storage in the latter.
```
➜ ls apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce*
apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.8.3.jar
apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.8.3.jar
apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.8.3.jar
apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.8.3.jar
apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.8.3.jar
➜ ls ~/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce*
/Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.8.3.jar
/Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.8.3.jar
/Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.8.3.jar
/Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.8.3.jar
```
That JAR is present in the hadoop-dependencies folder but as far as I can tell we don't include that folder in the classloader that's set up for the hadoop container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org