You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Aji Janis <aj...@gmail.com> on 2013/03/19 17:12:19 UTC
java.lang.NoSuchMethodError ThreadSafeClientConnManager
Hello,
I am getting the following syslog while running a mapreduce job on my
cluster:
2013-03-19 11:00:37,465 INFO
org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
client, config:
2013-03-19 11:00:37,477 INFO
org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
Initialized cache for UID to User mapping with a cache timeout of
14400 seconds.
2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO:
Got UserName hadoop for UID 8004 from the native
implementation*2013-03-19 11:00:37,554 FATAL
org.apache.hadoop.mapred.Child: Error running child :
java.lang.NoSuchMethodError:
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
<init>()V not found
at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
at org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)*
I think the issue is that somewhere I have a wrong version of the
httpclient jar (I need 4.0 or higher but I have a lower version somewhere).
I say this because the zookeeper classpath in syslog shows a library that I
can't find it in my cluster anywhere.
Syslog showed the zookeeper classpath as follows:
*2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path*=/opt/hadoop/bin/../conf:
/usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
/opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
/opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
/opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
/opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
/opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
/opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/*commons-httpclient-3.0.1.jar*:
/opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
/opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
/opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
/opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
/opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
/opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
/opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
/opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
/opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
/opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
/opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
/opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
/opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
/opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
/opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
but in all the nodes in my cluster we have removed commons-httpclient from
/opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
commons-httpclient is coming from. See below:
[root@mynode MYDIRNAME]# ls /opt/hadoop/lib
aspectjrt-1.6.5.jar commons-configuration-1.6.jar
commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
jsch-0.1.42.jar native
aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
oro-2.0.8.jar
commons-beanutils-1.7.0.jar commons-digester-1.8.jar
core-3.1.1.jar jasper-runtime-5.5.12.jar
junit-4.5.jar servlet-api-2.5-20081211.jar
commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
hsqldb-1.8.0.10.jar jdiff
kfs-0.2.2.jar slf4j-api-1.4.3.jar
commons-cli-1.2.jar commons-lang-2.4.jar
hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
commons-codec-1.4.jar commons-logging-1.1.1.jar
*httpclient-4.2.3.jar* jetty-6.1.26.jar
log4j-1.2.15.jar xmlenc-0.52.jar
commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
mockito-all-1.8.5.jar
Any suggestions? Does anyone know where the Zookeeper is getting the
classpath/library information? Do I need to restart my Zookeeper? Not sure
what the problem is. Any suggestions would be awesome. Thank you.
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
Depending on the version of Hadoop in use, you'd either set
mapreduce.task.classpath.user.precedence,
mapreduce.user.classpath.first or mapreduce.job.user.classpath.first
in your Job's configuration as true before submitting the job.
This feature/parameter is not available in the Apache Hadoop 0.20.2
version, however. It was added in later so you'll have better luck
trying a more recent version.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
It shouldn't be cached anywhere but try roll-restarting your
tasktrackers after having done the replacement.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
using hadoop 0.20.2 - FYI
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
using hadoop 0.20.2 - FYI
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
It shouldn't be cached anywhere but try roll-restarting your
tasktrackers after having done the replacement.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
using hadoop 0.20.2 - FYI
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
using hadoop 0.20.2 - FYI
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
Depending on the version of Hadoop in use, you'd either set
mapreduce.task.classpath.user.precedence,
mapreduce.user.classpath.first or mapreduce.job.user.classpath.first
in your Job's configuration as true before submitting the job.
This feature/parameter is not available in the Apache Hadoop 0.20.2
version, however. It was added in later so you'll have better luck
trying a more recent version.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
It shouldn't be cached anywhere but try roll-restarting your
tasktrackers after having done the replacement.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
It shouldn't be cached anywhere but try roll-restarting your
tasktrackers after having done the replacement.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
Depending on the version of Hadoop in use, you'd either set
mapreduce.task.classpath.user.precedence,
mapreduce.user.classpath.first or mapreduce.job.user.classpath.first
in your Job's configuration as true before submitting the job.
This feature/parameter is not available in the Apache Hadoop 0.20.2
version, however. It was added in later so you'll have better luck
trying a more recent version.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
Depending on the version of Hadoop in use, you'd either set
mapreduce.task.classpath.user.precedence,
mapreduce.user.classpath.first or mapreduce.job.user.classpath.first
in your Job's configuration as true before submitting the job.
This feature/parameter is not available in the Apache Hadoop 0.20.2
version, however. It was added in later so you'll have better luck
trying a more recent version.
On Tue, Mar 19, 2013 at 9:55 PM, Aji Janis <aj...@gmail.com> wrote:
> So I tried the replace commons-httpclient.jar with httpclient.jar on all
> nodes method and couldn't find a single node that had commons-httpclient
> which is why its very confusing that the zookeeper still shows that on the
> classpath.. like its cached somewhere? If it is, how do I flush it?
>
> But I like your option of:
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> ... How do I do this?
>
> Thanks Harsh.
>
>
> On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> ZK is showing its runtime JVM classpath (from the JVM that invoked a
>> ZK client), and not the ZK server's classpath. The path reported is
>> the below, which is part of /opt/hadoop/lib itself:
>>
>> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>>
>> Check for this jar's existence on all nodes and replace them with your
>> required version perhaps? I'm not sure if it will work but that seems
>> to be what you wish to try.
>>
>> I'd instead generally recommend passing the dependencies you need as
>> part of your job along with it via -libjars and such, and setting the
>> MR option to have it take precedence over the hadoop-provided jars of
>> the same kind.
>>
>> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am getting the following syslog while running a mapreduce job on my
>> > cluster:
>> >
>> > 2013-03-19 11:00:37,465 INFO
>> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
>> > client,
>> > config:
>> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
>> > Initialized cache for UID to User mapping with a cache timeout of 14400
>> > seconds.
>> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
>> > UserName hadoop for UID 8004 from the native implementation
>> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
>> > running
>> > child : java.lang.NoSuchMethodError:
>> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
>> > <init>()V not found
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
>> > at
>> >
>> > org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>> >
>> >
>> > I think the issue is that somewhere I have a wrong version of the
>> > httpclient
>> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
>> > this
>> > because the zookeeper classpath in syslog shows a library that I can't
>> > find
>> > it in my cluster anywhere.
>> >
>> > Syslog showed the zookeeper classpath as follows:
>> >
>> >
>> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
>> > environment:java.class.path=/opt/hadoop/bin/../conf:
>> >
>> > /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
>> >
>> > /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
>> >
>> > /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
>> >
>> > /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
>> >
>> > /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
>> >
>> > /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
>> >
>> > /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
>> >
>> > /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>> >
>> >
>> > but in all the nodes in my cluster we have removed commons-httpclient
>> > from
>> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
>> > commons-httpclient is coming from. See below:
>> >
>> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
>> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
>> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
>> > jsch-0.1.42.jar
>> > native
>> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
>> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
>> > oro-2.0.8.jar
>> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
>> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
>> > servlet-api-2.5-20081211.jar
>> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
>> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
>> > slf4j-api-1.4.3.jar
>> > commons-cli-1.2.jar commons-lang-2.4.jar
>> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
>> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
>> > commons-codec-1.4.jar commons-logging-1.1.1.jar
>> > httpclient-4.2.3.jar jetty-6.1.26.jar
>> > log4j-1.2.15.jar
>> > xmlenc-0.52.jar
>> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
>> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
>> > mockito-all-1.8.5.jar
>> >
>> >
>> >
>> > Any suggestions? Does anyone know where the Zookeeper is getting the
>> > classpath/library information? Do I need to restart my Zookeeper? Not
>> > sure
>> > what the problem is. Any suggestions would be awesome. Thank you.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
So I tried the replace commons-httpclient.jar with httpclient.jar on all
nodes method and couldn't find a single node that had commons-httpclient
which is why its very confusing that the zookeeper still shows that on the
classpath.. like its cached somewhere? If it is, how do I flush it?
But I like your option of:
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
... How do I do this?
Thanks Harsh.
On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
> ZK is showing its runtime JVM classpath (from the JVM that invoked a
> ZK client), and not the ZK server's classpath. The path reported is
> the below, which is part of /opt/hadoop/lib itself:
>
> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>
> Check for this jar's existence on all nodes and replace them with your
> required version perhaps? I'm not sure if it will work but that seems
> to be what you wish to try.
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> > Hello,
> >
> > I am getting the following syslog while running a mapreduce job on my
> > cluster:
> >
> > 2013-03-19 11:00:37,465 INFO
> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
> client,
> > config:
> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> > Initialized cache for UID to User mapping with a cache timeout of 14400
> > seconds.
> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> > UserName hadoop for UID 8004 from the native implementation
> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
> running
> > child : java.lang.NoSuchMethodError:
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> > <init>()V not found
> > at
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> > at
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
> >
> >
> > I think the issue is that somewhere I have a wrong version of the
> httpclient
> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
> this
> > because the zookeeper classpath in syslog shows a library that I can't
> find
> > it in my cluster anywhere.
> >
> > Syslog showed the zookeeper classpath as follows:
> >
> >
> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/hadoop/bin/../conf:
> >
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> >
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> >
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> >
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> >
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> >
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> >
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> >
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> >
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> >
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> >
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> >
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> >
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> >
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> >
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> >
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
> >
> >
> > but in all the nodes in my cluster we have removed commons-httpclient
> from
> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> > commons-httpclient is coming from. See below:
> >
> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
> jsch-0.1.42.jar
> > native
> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> > oro-2.0.8.jar
> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> > servlet-api-2.5-20081211.jar
> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> > slf4j-api-1.4.3.jar
> > commons-cli-1.2.jar commons-lang-2.4.jar
> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> > commons-codec-1.4.jar commons-logging-1.1.1.jar
> > httpclient-4.2.3.jar jetty-6.1.26.jar
> log4j-1.2.15.jar
> > xmlenc-0.52.jar
> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> > mockito-all-1.8.5.jar
> >
> >
> >
> > Any suggestions? Does anyone know where the Zookeeper is getting the
> > classpath/library information? Do I need to restart my Zookeeper? Not
> sure
> > what the problem is. Any suggestions would be awesome. Thank you.
> >
> >
>
>
>
> --
> Harsh J
>
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
So I tried the replace commons-httpclient.jar with httpclient.jar on all
nodes method and couldn't find a single node that had commons-httpclient
which is why its very confusing that the zookeeper still shows that on the
classpath.. like its cached somewhere? If it is, how do I flush it?
But I like your option of:
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
... How do I do this?
Thanks Harsh.
On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
> ZK is showing its runtime JVM classpath (from the JVM that invoked a
> ZK client), and not the ZK server's classpath. The path reported is
> the below, which is part of /opt/hadoop/lib itself:
>
> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>
> Check for this jar's existence on all nodes and replace them with your
> required version perhaps? I'm not sure if it will work but that seems
> to be what you wish to try.
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> > Hello,
> >
> > I am getting the following syslog while running a mapreduce job on my
> > cluster:
> >
> > 2013-03-19 11:00:37,465 INFO
> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
> client,
> > config:
> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> > Initialized cache for UID to User mapping with a cache timeout of 14400
> > seconds.
> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> > UserName hadoop for UID 8004 from the native implementation
> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
> running
> > child : java.lang.NoSuchMethodError:
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> > <init>()V not found
> > at
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> > at
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
> >
> >
> > I think the issue is that somewhere I have a wrong version of the
> httpclient
> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
> this
> > because the zookeeper classpath in syslog shows a library that I can't
> find
> > it in my cluster anywhere.
> >
> > Syslog showed the zookeeper classpath as follows:
> >
> >
> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/hadoop/bin/../conf:
> >
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> >
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> >
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> >
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> >
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> >
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> >
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> >
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> >
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> >
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> >
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> >
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> >
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> >
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> >
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> >
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
> >
> >
> > but in all the nodes in my cluster we have removed commons-httpclient
> from
> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> > commons-httpclient is coming from. See below:
> >
> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
> jsch-0.1.42.jar
> > native
> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> > oro-2.0.8.jar
> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> > servlet-api-2.5-20081211.jar
> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> > slf4j-api-1.4.3.jar
> > commons-cli-1.2.jar commons-lang-2.4.jar
> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> > commons-codec-1.4.jar commons-logging-1.1.1.jar
> > httpclient-4.2.3.jar jetty-6.1.26.jar
> log4j-1.2.15.jar
> > xmlenc-0.52.jar
> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> > mockito-all-1.8.5.jar
> >
> >
> >
> > Any suggestions? Does anyone know where the Zookeeper is getting the
> > classpath/library information? Do I need to restart my Zookeeper? Not
> sure
> > what the problem is. Any suggestions would be awesome. Thank you.
> >
> >
>
>
>
> --
> Harsh J
>
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
So I tried the replace commons-httpclient.jar with httpclient.jar on all
nodes method and couldn't find a single node that had commons-httpclient
which is why its very confusing that the zookeeper still shows that on the
classpath.. like its cached somewhere? If it is, how do I flush it?
But I like your option of:
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
... How do I do this?
Thanks Harsh.
On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
> ZK is showing its runtime JVM classpath (from the JVM that invoked a
> ZK client), and not the ZK server's classpath. The path reported is
> the below, which is part of /opt/hadoop/lib itself:
>
> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>
> Check for this jar's existence on all nodes and replace them with your
> required version perhaps? I'm not sure if it will work but that seems
> to be what you wish to try.
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> > Hello,
> >
> > I am getting the following syslog while running a mapreduce job on my
> > cluster:
> >
> > 2013-03-19 11:00:37,465 INFO
> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
> client,
> > config:
> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> > Initialized cache for UID to User mapping with a cache timeout of 14400
> > seconds.
> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> > UserName hadoop for UID 8004 from the native implementation
> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
> running
> > child : java.lang.NoSuchMethodError:
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> > <init>()V not found
> > at
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> > at
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
> >
> >
> > I think the issue is that somewhere I have a wrong version of the
> httpclient
> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
> this
> > because the zookeeper classpath in syslog shows a library that I can't
> find
> > it in my cluster anywhere.
> >
> > Syslog showed the zookeeper classpath as follows:
> >
> >
> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/hadoop/bin/../conf:
> >
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> >
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> >
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> >
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> >
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> >
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> >
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> >
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> >
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> >
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> >
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> >
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> >
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> >
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> >
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> >
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
> >
> >
> > but in all the nodes in my cluster we have removed commons-httpclient
> from
> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> > commons-httpclient is coming from. See below:
> >
> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
> jsch-0.1.42.jar
> > native
> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> > oro-2.0.8.jar
> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> > servlet-api-2.5-20081211.jar
> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> > slf4j-api-1.4.3.jar
> > commons-cli-1.2.jar commons-lang-2.4.jar
> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> > commons-codec-1.4.jar commons-logging-1.1.1.jar
> > httpclient-4.2.3.jar jetty-6.1.26.jar
> log4j-1.2.15.jar
> > xmlenc-0.52.jar
> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> > mockito-all-1.8.5.jar
> >
> >
> >
> > Any suggestions? Does anyone know where the Zookeeper is getting the
> > classpath/library information? Do I need to restart my Zookeeper? Not
> sure
> > what the problem is. Any suggestions would be awesome. Thank you.
> >
> >
>
>
>
> --
> Harsh J
>
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Aji Janis <aj...@gmail.com>.
So I tried the replace commons-httpclient.jar with httpclient.jar on all
nodes method and couldn't find a single node that had commons-httpclient
which is why its very confusing that the zookeeper still shows that on the
classpath.. like its cached somewhere? If it is, how do I flush it?
But I like your option of:
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
... How do I do this?
Thanks Harsh.
On Tue, Mar 19, 2013 at 12:19 PM, Harsh J <ha...@cloudera.com> wrote:
> ZK is showing its runtime JVM classpath (from the JVM that invoked a
> ZK client), and not the ZK server's classpath. The path reported is
> the below, which is part of /opt/hadoop/lib itself:
>
> /opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
>
> Check for this jar's existence on all nodes and replace them with your
> required version perhaps? I'm not sure if it will work but that seems
> to be what you wish to try.
>
> I'd instead generally recommend passing the dependencies you need as
> part of your job along with it via -libjars and such, and setting the
> MR option to have it take precedence over the hadoop-provided jars of
> the same kind.
>
> On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> > Hello,
> >
> > I am getting the following syslog while running a mapreduce job on my
> > cluster:
> >
> > 2013-03-19 11:00:37,465 INFO
> > org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http
> client,
> > config:
> > 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> > Initialized cache for UID to User mapping with a cache timeout of 14400
> > seconds.
> > 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> > UserName hadoop for UID 8004 from the native implementation
> > 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error
> running
> > child : java.lang.NoSuchMethodError:
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> > <init>()V not found
> > at
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> > at
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
> >
> >
> > I think the issue is that somewhere I have a wrong version of the
> httpclient
> > jar (I need 4.0 or higher but I have a lower version somewhere). I say
> this
> > because the zookeeper classpath in syslog shows a library that I can't
> find
> > it in my cluster anywhere.
> >
> > Syslog showed the zookeeper classpath as follows:
> >
> >
> > 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/hadoop/bin/../conf:
> >
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> >
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> >
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> >
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> >
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> >
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> >
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> >
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> >
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> >
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> >
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> >
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> >
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> >
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> >
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> >
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> >
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> >
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
> >
> >
> > but in all the nodes in my cluster we have removed commons-httpclient
> from
> > /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> > commons-httpclient is coming from. See below:
> >
> > [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> > aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> > commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar
> jsch-0.1.42.jar
> > native
> > aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> > commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> > oro-2.0.8.jar
> > commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> > core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> > servlet-api-2.5-20081211.jar
> > commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> > hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> > slf4j-api-1.4.3.jar
> > commons-cli-1.2.jar commons-lang-2.4.jar
> > hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> > kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> > commons-codec-1.4.jar commons-logging-1.1.1.jar
> > httpclient-4.2.3.jar jetty-6.1.26.jar
> log4j-1.2.15.jar
> > xmlenc-0.52.jar
> > commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> > jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> > mockito-all-1.8.5.jar
> >
> >
> >
> > Any suggestions? Does anyone know where the Zookeeper is getting the
> > classpath/library information? Do I need to restart my Zookeeper? Not
> sure
> > what the problem is. Any suggestions would be awesome. Thank you.
> >
> >
>
>
>
> --
> Harsh J
>
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
ZK is showing its runtime JVM classpath (from the JVM that invoked a
ZK client), and not the ZK server's classpath. The path reported is
the below, which is part of /opt/hadoop/lib itself:
/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
Check for this jar's existence on all nodes and replace them with your
required version perhaps? I'm not sure if it will work but that seems
to be what you wish to try.
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> Hello,
>
> I am getting the following syslog while running a mapreduce job on my
> cluster:
>
> 2013-03-19 11:00:37,465 INFO
> org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http client,
> config:
> 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of 14400
> seconds.
> 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> UserName hadoop for UID 8004 from the native implementation
> 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> <init>()V not found
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> at
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>
>
> I think the issue is that somewhere I have a wrong version of the httpclient
> jar (I need 4.0 or higher but I have a lower version somewhere). I say this
> because the zookeeper classpath in syslog shows a library that I can't find
> it in my cluster anywhere.
>
> Syslog showed the zookeeper classpath as follows:
>
>
> 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hadoop/bin/../conf:
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>
>
> but in all the nodes in my cluster we have removed commons-httpclient from
> /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> commons-httpclient is coming from. See below:
>
> [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar jsch-0.1.42.jar
> native
> aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> oro-2.0.8.jar
> commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> servlet-api-2.5-20081211.jar
> commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> slf4j-api-1.4.3.jar
> commons-cli-1.2.jar commons-lang-2.4.jar
> hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> commons-codec-1.4.jar commons-logging-1.1.1.jar
> httpclient-4.2.3.jar jetty-6.1.26.jar log4j-1.2.15.jar
> xmlenc-0.52.jar
> commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> mockito-all-1.8.5.jar
>
>
>
> Any suggestions? Does anyone know where the Zookeeper is getting the
> classpath/library information? Do I need to restart my Zookeeper? Not sure
> what the problem is. Any suggestions would be awesome. Thank you.
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
ZK is showing its runtime JVM classpath (from the JVM that invoked a
ZK client), and not the ZK server's classpath. The path reported is
the below, which is part of /opt/hadoop/lib itself:
/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
Check for this jar's existence on all nodes and replace them with your
required version perhaps? I'm not sure if it will work but that seems
to be what you wish to try.
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> Hello,
>
> I am getting the following syslog while running a mapreduce job on my
> cluster:
>
> 2013-03-19 11:00:37,465 INFO
> org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http client,
> config:
> 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of 14400
> seconds.
> 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> UserName hadoop for UID 8004 from the native implementation
> 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> <init>()V not found
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> at
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>
>
> I think the issue is that somewhere I have a wrong version of the httpclient
> jar (I need 4.0 or higher but I have a lower version somewhere). I say this
> because the zookeeper classpath in syslog shows a library that I can't find
> it in my cluster anywhere.
>
> Syslog showed the zookeeper classpath as follows:
>
>
> 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hadoop/bin/../conf:
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>
>
> but in all the nodes in my cluster we have removed commons-httpclient from
> /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> commons-httpclient is coming from. See below:
>
> [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar jsch-0.1.42.jar
> native
> aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> oro-2.0.8.jar
> commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> servlet-api-2.5-20081211.jar
> commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> slf4j-api-1.4.3.jar
> commons-cli-1.2.jar commons-lang-2.4.jar
> hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> commons-codec-1.4.jar commons-logging-1.1.1.jar
> httpclient-4.2.3.jar jetty-6.1.26.jar log4j-1.2.15.jar
> xmlenc-0.52.jar
> commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> mockito-all-1.8.5.jar
>
>
>
> Any suggestions? Does anyone know where the Zookeeper is getting the
> classpath/library information? Do I need to restart my Zookeeper? Not sure
> what the problem is. Any suggestions would be awesome. Thank you.
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
ZK is showing its runtime JVM classpath (from the JVM that invoked a
ZK client), and not the ZK server's classpath. The path reported is
the below, which is part of /opt/hadoop/lib itself:
/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
Check for this jar's existence on all nodes and replace them with your
required version perhaps? I'm not sure if it will work but that seems
to be what you wish to try.
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> Hello,
>
> I am getting the following syslog while running a mapreduce job on my
> cluster:
>
> 2013-03-19 11:00:37,465 INFO
> org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http client,
> config:
> 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of 14400
> seconds.
> 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> UserName hadoop for UID 8004 from the native implementation
> 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> <init>()V not found
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> at
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>
>
> I think the issue is that somewhere I have a wrong version of the httpclient
> jar (I need 4.0 or higher but I have a lower version somewhere). I say this
> because the zookeeper classpath in syslog shows a library that I can't find
> it in my cluster anywhere.
>
> Syslog showed the zookeeper classpath as follows:
>
>
> 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hadoop/bin/../conf:
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>
>
> but in all the nodes in my cluster we have removed commons-httpclient from
> /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> commons-httpclient is coming from. See below:
>
> [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar jsch-0.1.42.jar
> native
> aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> oro-2.0.8.jar
> commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> servlet-api-2.5-20081211.jar
> commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> slf4j-api-1.4.3.jar
> commons-cli-1.2.jar commons-lang-2.4.jar
> hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> commons-codec-1.4.jar commons-logging-1.1.1.jar
> httpclient-4.2.3.jar jetty-6.1.26.jar log4j-1.2.15.jar
> xmlenc-0.52.jar
> commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> mockito-all-1.8.5.jar
>
>
>
> Any suggestions? Does anyone know where the Zookeeper is getting the
> classpath/library information? Do I need to restart my Zookeeper? Not sure
> what the problem is. Any suggestions would be awesome. Thank you.
>
>
--
Harsh J
Re: java.lang.NoSuchMethodError ThreadSafeClientConnManager
Posted by Harsh J <ha...@cloudera.com>.
ZK is showing its runtime JVM classpath (from the JVM that invoked a
ZK client), and not the ZK server's classpath. The path reported is
the below, which is part of /opt/hadoop/lib itself:
/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar
Check for this jar's existence on all nodes and replace them with your
required version perhaps? I'm not sure if it will work but that seems
to be what you wish to try.
I'd instead generally recommend passing the dependencies you need as
part of your job along with it via -libjars and such, and setting the
MR option to have it take precedence over the hadoop-provided jars of
the same kind.
On Tue, Mar 19, 2013 at 9:42 PM, Aji Janis <aj...@gmail.com> wrote:
> Hello,
>
> I am getting the following syslog while running a mapreduce job on my
> cluster:
>
> 2013-03-19 11:00:37,465 INFO
> org.apache.solr.client.solrj.fimpl.HttpClientUtil: Creating new http client,
> config:
> 2013-03-19 11:00:37,477 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-03-19 11:00:37,550 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of 14400
> seconds.
> 2013-03-19 11:00:37,551 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
> UserName hadoop for UID 8004 from the native implementation
> 2013-03-19 11:00:37,554 FATAL org.apache.hadoop.mapred.Child: Error running
> child : java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method
> <init>()V not found
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:103)
> at
> org.apache.solr.client.solrj.impl.CloudSolrServer.<init>(CloudSolrServer.java:83)
>
>
> I think the issue is that somewhere I have a wrong version of the httpclient
> jar (I need 4.0 or higher but I have a lower version somewhere). I say this
> because the zookeeper classpath in syslog shows a library that I can't find
> it in my cluster anywhere.
>
> Syslog showed the zookeeper classpath as follows:
>
>
> 2013-03-19 11:00:22,347 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hadoop/bin/../conf:
> /usr/java/default/lib/tools.jar:/opt/hadoop/bin/..:/opt/hadoop/bin/../hadoop-core-0.20.203.0.jar:
> /opt/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/bin/../lib/aspectjtools-1.6.5.jar:
> /opt/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:
> /opt/hadoop/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.4.jar:
> /opt/hadoop/bin/../lib/commons-collections-3.2.1.jar:/opt/hadoop/bin/../lib/commons-configuration-1.6.jar:
> /opt/hadoop/bin/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/bin/../lib/commons-digester-1.8.jar:
> /opt/hadoop/bin/../lib/commons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:
> /opt/hadoop/bin/../lib/commons-lang-2.4.jar:/opt/hadoop/bin/../lib/commons-logging-1.1.1.jar:
> /opt/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/bin/../lib/commons-math-2.1.jar:
> /opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/bin/../lib/core-3.1.1.jar:
> /opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/bin/../lib/jackson-core-asl-1.0.1.jar:
> /opt/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:
> /opt/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:
> /opt/hadoop/bin/../lib/jetty-6.1.26.jar:/opt/hadoop/bin/../lib/jetty-util-6.1.26.jar:
> /opt/hadoop/bin/../lib/jsch-0.1.42.jar:/opt/hadoop/bin/../lib/junit-4.5.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:
> /opt/hadoop/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.5.jar:/opt/hadoop/bin/../lib/oro-2.0.8.jar:
> /opt/hadoop/bin/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:
> /opt/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:
> /opt/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:./:/conf:/build/*:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars/classes:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/jars:
> /opt/hadoop-data/mapred/local/taskTracker/MYDIRNAME/jobcache/job_201301171136_1906/attempt_201301171136_1906_m_000000_0/work
>
>
> but in all the nodes in my cluster we have removed commons-httpclient from
> /opt/hadoop/lib and put in httpclient-4.2.3.jar so I am not sure where
> commons-httpclient is coming from. See below:
>
> [root@mynode MYDIRNAME]# ls /opt/hadoop/lib
> aspectjrt-1.6.5.jar commons-configuration-1.6.jar
> commons-math-2.1.jar jackson-mapper-asl-1.0.1.jar jsch-0.1.42.jar
> native
> aspectjtools-1.6.5.jar commons-daemon-1.0.1.jar
> commons-net-1.4.1.jar jasper-compiler-5.5.12.jar jsp-2.1
> oro-2.0.8.jar
> commons-beanutils-1.7.0.jar commons-digester-1.8.jar
> core-3.1.1.jar jasper-runtime-5.5.12.jar junit-4.5.jar
> servlet-api-2.5-20081211.jar
> commons-beanutils-core-1.8.0.jar commons-el-1.0.jar
> hsqldb-1.8.0.10.jar jdiff kfs-0.2.2.jar
> slf4j-api-1.4.3.jar
> commons-cli-1.2.jar commons-lang-2.4.jar
> hsqldb-1.8.0.10.LICENSE.txt jets3t-0.6.1.jar
> kfs-0.2.LICENSE.txt slf4j-log4j12-1.4.3.jar
> commons-codec-1.4.jar commons-logging-1.1.1.jar
> httpclient-4.2.3.jar jetty-6.1.26.jar log4j-1.2.15.jar
> xmlenc-0.52.jar
> commons-collections-3.2.1.jar commons-logging-api-1.0.4.jar
> jackson-core-asl-1.0.1.jar jetty-util-6.1.26.jar
> mockito-all-1.8.5.jar
>
>
>
> Any suggestions? Does anyone know where the Zookeeper is getting the
> classpath/library information? Do I need to restart my Zookeeper? Not sure
> what the problem is. Any suggestions would be awesome. Thank you.
>
>
--
Harsh J