You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Sampath Herga <sa...@gmail.com> on 2012/12/05 12:53:42 UTC
Issue with third party library
Hi,
I was just trying to setup hadoop and run some sample programs. I ran into
some issues when trying to add a required library. I tried using the
DistributedCache methods to try adding to the classpath.
The basic code is:
Path mysqlJar = new
Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
fs.copyFromLocalFile(mysqlJar, mysqlJar);
DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
I did a copyFromLocal since I wasnt sure where it would get picked from. I
tried both addArchiveToClassPath and addFileToClassPath. But with both, I
get a ClassNotFoundException.
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.mysql.jdbc.Driver
at
org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
... 16 more
The job seems to have got the file added and also checked in the
tasktracker local dir and the file seems to be present.
<property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
Any clues would be helpful. I dont think I can use -libjars since I am
trying to execute the hadoop jobs from within another application and not
from the command line.
Regards,
Sampath.
Re: Issue with third party library
Posted by Sampath Herga <sa...@gmail.com>.
Hi Hemanth,
Setting the full path worked.
Thanks,
Sampath.
On Thu, Dec 6, 2012 at 9:51 AM, Hemanth Yamijala
<yh...@thoughtworks.com>wrote:
> Sampath,
>
> You mentioned that the file is present in the tasktracker local dir,
> could you please tell us the full path ? I am wondering if setting the full
> path will have any impact, rather than specifying the relative path.
>
> Another option may be to try to use the addCacheArchive and createSymLink
> APIs as described here:
> http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
>
>
> On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> I was just trying to setup hadoop and run some sample programs. I ran
>> into some issues when trying to add a required library. I tried using the
>> DistributedCache methods to try adding to the classpath.
>>
>> The basic code is:
>> Path mysqlJar = new
>> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
>> fs.copyFromLocalFile(mysqlJar, mysqlJar);
>> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>>
>> I did a copyFromLocal since I wasnt sure where it would get picked from.
>> I tried both addArchiveToClassPath and addFileToClassPath. But with both, I
>> get a ClassNotFoundException.
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> com.mysql.jdbc.Driver
>> at
>> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
>> ... 16 more
>>
>> The job seems to have got the file added and also checked in the
>> tasktracker local dir and the file seems to be present.
>>
>>
>> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>>
>> Any clues would be helpful. I dont think I can use -libjars since I am
>> trying to execute the hadoop jobs from within another application and not
>> from the command line.
>>
>> Regards,
>> Sampath.
>>
>
>
Re: Issue with third party library
Posted by Sampath Herga <sa...@gmail.com>.
Hi Hemanth,
Setting the full path worked.
Thanks,
Sampath.
On Thu, Dec 6, 2012 at 9:51 AM, Hemanth Yamijala
<yh...@thoughtworks.com>wrote:
> Sampath,
>
> You mentioned that the file is present in the tasktracker local dir,
> could you please tell us the full path ? I am wondering if setting the full
> path will have any impact, rather than specifying the relative path.
>
> Another option may be to try to use the addCacheArchive and createSymLink
> APIs as described here:
> http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
>
>
> On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> I was just trying to setup hadoop and run some sample programs. I ran
>> into some issues when trying to add a required library. I tried using the
>> DistributedCache methods to try adding to the classpath.
>>
>> The basic code is:
>> Path mysqlJar = new
>> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
>> fs.copyFromLocalFile(mysqlJar, mysqlJar);
>> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>>
>> I did a copyFromLocal since I wasnt sure where it would get picked from.
>> I tried both addArchiveToClassPath and addFileToClassPath. But with both, I
>> get a ClassNotFoundException.
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> com.mysql.jdbc.Driver
>> at
>> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
>> ... 16 more
>>
>> The job seems to have got the file added and also checked in the
>> tasktracker local dir and the file seems to be present.
>>
>>
>> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>>
>> Any clues would be helpful. I dont think I can use -libjars since I am
>> trying to execute the hadoop jobs from within another application and not
>> from the command line.
>>
>> Regards,
>> Sampath.
>>
>
>
Re: Issue with third party library
Posted by Sampath Herga <sa...@gmail.com>.
Hi Hemanth,
Setting the full path worked.
Thanks,
Sampath.
On Thu, Dec 6, 2012 at 9:51 AM, Hemanth Yamijala
<yh...@thoughtworks.com>wrote:
> Sampath,
>
> You mentioned that the file is present in the tasktracker local dir,
> could you please tell us the full path ? I am wondering if setting the full
> path will have any impact, rather than specifying the relative path.
>
> Another option may be to try to use the addCacheArchive and createSymLink
> APIs as described here:
> http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
>
>
> On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> I was just trying to setup hadoop and run some sample programs. I ran
>> into some issues when trying to add a required library. I tried using the
>> DistributedCache methods to try adding to the classpath.
>>
>> The basic code is:
>> Path mysqlJar = new
>> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
>> fs.copyFromLocalFile(mysqlJar, mysqlJar);
>> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>>
>> I did a copyFromLocal since I wasnt sure where it would get picked from.
>> I tried both addArchiveToClassPath and addFileToClassPath. But with both, I
>> get a ClassNotFoundException.
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> com.mysql.jdbc.Driver
>> at
>> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
>> ... 16 more
>>
>> The job seems to have got the file added and also checked in the
>> tasktracker local dir and the file seems to be present.
>>
>>
>> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>>
>> Any clues would be helpful. I dont think I can use -libjars since I am
>> trying to execute the hadoop jobs from within another application and not
>> from the command line.
>>
>> Regards,
>> Sampath.
>>
>
>
Re: Issue with third party library
Posted by Sampath Herga <sa...@gmail.com>.
Hi Hemanth,
Setting the full path worked.
Thanks,
Sampath.
On Thu, Dec 6, 2012 at 9:51 AM, Hemanth Yamijala
<yh...@thoughtworks.com>wrote:
> Sampath,
>
> You mentioned that the file is present in the tasktracker local dir,
> could you please tell us the full path ? I am wondering if setting the full
> path will have any impact, rather than specifying the relative path.
>
> Another option may be to try to use the addCacheArchive and createSymLink
> APIs as described here:
> http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
>
>
> On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> I was just trying to setup hadoop and run some sample programs. I ran
>> into some issues when trying to add a required library. I tried using the
>> DistributedCache methods to try adding to the classpath.
>>
>> The basic code is:
>> Path mysqlJar = new
>> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
>> fs.copyFromLocalFile(mysqlJar, mysqlJar);
>> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>>
>> I did a copyFromLocal since I wasnt sure where it would get picked from.
>> I tried both addArchiveToClassPath and addFileToClassPath. But with both, I
>> get a ClassNotFoundException.
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> com.mysql.jdbc.Driver
>> at
>> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
>> ... 16 more
>>
>> The job seems to have got the file added and also checked in the
>> tasktracker local dir and the file seems to be present.
>>
>>
>> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>>
>> Any clues would be helpful. I dont think I can use -libjars since I am
>> trying to execute the hadoop jobs from within another application and not
>> from the command line.
>>
>> Regards,
>> Sampath.
>>
>
>
Re: Issue with third party library
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Sampath,
You mentioned that the file is present in the tasktracker local dir, could
you please tell us the full path ? I am wondering if setting the full path
will have any impact, rather than specifying the relative path.
Another option may be to try to use the addCacheArchive and createSymLink
APIs as described here:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
> Hi,
>
> I was just trying to setup hadoop and run some sample programs. I ran into
> some issues when trying to add a required library. I tried using the
> DistributedCache methods to try adding to the classpath.
>
> The basic code is:
> Path mysqlJar = new
> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
> fs.copyFromLocalFile(mysqlJar, mysqlJar);
> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>
> I did a copyFromLocal since I wasnt sure where it would get picked from. I
> tried both addArchiveToClassPath and addFileToClassPath. But with both, I
> get a ClassNotFoundException.
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
> at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
> ... 16 more
>
> The job seems to have got the file added and also checked in the
> tasktracker local dir and the file seems to be present.
>
>
> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>
> Any clues would be helpful. I dont think I can use -libjars since I am
> trying to execute the hadoop jobs from within another application and not
> from the command line.
>
> Regards,
> Sampath.
>
Re: Issue with third party library
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Sampath,
You mentioned that the file is present in the tasktracker local dir, could
you please tell us the full path ? I am wondering if setting the full path
will have any impact, rather than specifying the relative path.
Another option may be to try to use the addCacheArchive and createSymLink
APIs as described here:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
> Hi,
>
> I was just trying to setup hadoop and run some sample programs. I ran into
> some issues when trying to add a required library. I tried using the
> DistributedCache methods to try adding to the classpath.
>
> The basic code is:
> Path mysqlJar = new
> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
> fs.copyFromLocalFile(mysqlJar, mysqlJar);
> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>
> I did a copyFromLocal since I wasnt sure where it would get picked from. I
> tried both addArchiveToClassPath and addFileToClassPath. But with both, I
> get a ClassNotFoundException.
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
> at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
> ... 16 more
>
> The job seems to have got the file added and also checked in the
> tasktracker local dir and the file seems to be present.
>
>
> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>
> Any clues would be helpful. I dont think I can use -libjars since I am
> trying to execute the hadoop jobs from within another application and not
> from the command line.
>
> Regards,
> Sampath.
>
Re: Issue with third party library
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Sampath,
You mentioned that the file is present in the tasktracker local dir, could
you please tell us the full path ? I am wondering if setting the full path
will have any impact, rather than specifying the relative path.
Another option may be to try to use the addCacheArchive and createSymLink
APIs as described here:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
> Hi,
>
> I was just trying to setup hadoop and run some sample programs. I ran into
> some issues when trying to add a required library. I tried using the
> DistributedCache methods to try adding to the classpath.
>
> The basic code is:
> Path mysqlJar = new
> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
> fs.copyFromLocalFile(mysqlJar, mysqlJar);
> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>
> I did a copyFromLocal since I wasnt sure where it would get picked from. I
> tried both addArchiveToClassPath and addFileToClassPath. But with both, I
> get a ClassNotFoundException.
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
> at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
> ... 16 more
>
> The job seems to have got the file added and also checked in the
> tasktracker local dir and the file seems to be present.
>
>
> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>
> Any clues would be helpful. I dont think I can use -libjars since I am
> trying to execute the hadoop jobs from within another application and not
> from the command line.
>
> Regards,
> Sampath.
>
Re: Issue with third party library
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Sampath,
You mentioned that the file is present in the tasktracker local dir, could
you please tell us the full path ? I am wondering if setting the full path
will have any impact, rather than specifying the relative path.
Another option may be to try to use the addCacheArchive and createSymLink
APIs as described here:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#DistributedCache
On Wed, Dec 5, 2012 at 5:23 PM, Sampath Herga <sa...@gmail.com> wrote:
> Hi,
>
> I was just trying to setup hadoop and run some sample programs. I ran into
> some issues when trying to add a required library. I tried using the
> DistributedCache methods to try adding to the classpath.
>
> The basic code is:
> Path mysqlJar = new
> Path("target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar");
> fs.copyFromLocalFile(mysqlJar, mysqlJar);
> DistributedCache.addArchiveToClassPath(mysqlJar, jobConf, fs);
>
> I did a copyFromLocal since I wasnt sure where it would get picked from. I
> tried both addArchiveToClassPath and addFileToClassPath. But with both, I
> get a ClassNotFoundException.
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
> at
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271)
> ... 16 more
>
> The job seems to have got the file added and also checked in the
> tasktracker local dir and the file seems to be present.
>
>
> <property><name>mapred.job.classpath.archives</name><value>target/classes/META-INF/mysql-connector-java-3.1.12-bin.jar</value></property>
>
> Any clues would be helpful. I dont think I can use -libjars since I am
> trying to execute the hadoop jobs from within another application and not
> from the command line.
>
> Regards,
> Sampath.
>