You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by chelikani narasimharao <na...@gmail.com> on 2012/02/16 15:54:15 UTC
need help on merge tool in sqoop
Hi All,
I am trying to merge the two datasets using the merge tool
functionality in sqoop.But i got struck up with the following error.
*java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
*
*---->below is my command executed to get the first data set specifying the
location to store the first dataset to dataset1 under narasimharao directory
*
sqoop import --connect jdbc:hpt4jdbc://
g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
com.hp.t4jdbc.HPT4Driver --username narasimharoa@hp.com -P --verbose
--split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and CNTCT_ID
in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
--target-dir /home/narasimharao/dataset1
successfully able to import the dataset1 and also the .class and .jar files
were written to the following location
*/tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2*
*--->below is my command executed to get the second data set after
updating the few records which exists in the dataset1 with the same
merge-key*
sqoop import --connect jdbc:hpt4jdbc://
g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
com.hp.t4jdbc.HPT4Driver --username narasimharoa.chelikani@hp.com -P
--verbose --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and
CNTCT_ID in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
--target-dir /home/narasimharao/dataset2
successfully able to import the dataset1 and also the .class and .jar files
were written to the following location
*
/tmp/sqoop-narasimharao/compile/e26282f14a672e7722352fb424781e5a/CNTCT_D.jar
*
*--->Now the actually Sqoop merge command was given like below:*
*sqoop merge --new-data /home/narasimharao/dataset1 --onto
/home/narasimharao/dataset2 --target-dir /home/narasimharao/merged
--jar-file CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID*
Can you please help me which class name i need to mention over here under
the class-name attribute and what's wrong in this statement.
I tried the below options
1.copied the dataset1 corresponding class and jar files into my home
direcotry and tried like below...but getting the same error as below
sqoop merge --new-data /home/narasimharao/dataset2 --onto
/home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
--jar-file ./CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
12/02/16 13:56:08 INFO input.FileInputFormat: Total input paths to process
: 2
12/02/16 13:56:08 WARN snappy.LoadSnappy: Snappy native library is available
12/02/16 13:56:08 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
12/02/16 13:56:08 INFO snappy.LoadSnappy: Snappy native library loaded
12/02/16 13:56:10 INFO mapred.JobClient: Running job: job_201202161202_0005
12/02/16 13:56:11 INFO mapred.JobClient: map 0% reduce 0%
12/02/16 13:56:55 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000000_0, Status : FAILED
*java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
* at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
12/02/16 13:56:59 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000001_0, Status : FAILED
*java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
* at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
12/02/16 13:57:22 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
12/02/16 13:57:23 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000001_1, Status : FAILED
*java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
* at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
12/02/16 13:57:50 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000000_2, Status : FAILED
*java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
* at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
12/02/16 13:57:51 INFO mapred.JobClient: Task Id :
attempt_201202161202_0005_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLo
attempt_201202161202_0005_m_000001_2: log4j:WARN No appenders could be
found for logger (org.apache.hadoop.mapred.Task).
attempt_201202161202_0005_m_000001_2: log4j:WARN Please initialize the
log4j system properly.
12/02/16 13:58:27 INFO mapred.JobClient: Job complete: job_201202161202_0005
12/02/16 13:58:28 INFO mapred.JobClient: Counters: 7
12/02/16 13:58:28 INFO mapred.JobClient: Job Counters
12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=249623
12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
12/02/16 13:58:28 INFO mapred.JobClient: Launched map tasks=8
12/02/16 13:58:28 INFO mapred.JobClient: Data-local map tasks=8
12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
12/02/16 13:58:28 INFO mapred.JobClient: Failed map tasks=1
12/02/16 13:58:28 ERROR tool.MergeTool: MapReduce job failed!
2. given second import dataset2 corresponding classname..but the same error
it is also throwing
3.mentioned with full path for both .class and .jar file
sqoop merge --new-data /home/narasimharao/dataset2 --onto
/home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
--jar-file
/tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.jar
--class-name
/tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.class
--merge-key CNTCT_ID
4.tried the above stmt3 with second dataset .class and .jar files
also..getting the same errors
Any Help Appreciated!!!!!!
Thanks,
Narasimharao
Re: need help on merge tool in sqoop
Posted by Arvind Prabhakar <ar...@apache.org>.
Hi Narasimharao,
When you invoke the merge command, try specifying the complete path of the
jar file to use. From the errors you have described, it appears that Sqoop
is unable to load the jar file correctly.
Thanks,
Arvind
On Tue, Feb 21, 2012 at 8:28 PM, chelikani narasimharao <
narasimharaoc@gmail.com> wrote:
> hi guys,
> Can any one please help me for the below issue.
>
> Advanced Thanks,
> Narasimharao
>
>
> On Thu, Feb 16, 2012 at 8:24 PM, chelikani narasimharao <
> narasimharaoc@gmail.com> wrote:
>
>> Hi All,
>>
>> I am trying to merge the two datasets using the merge tool
>> functionality in sqoop.But i got struck up with the following error.
>>
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class*
>>
>> *---->below is my command executed to get the first data set specifying
>> the location to store the first dataset to dataset1 under narasimharao
>> directory*
>>
>> sqoop import --connect jdbc:hpt4jdbc://
>> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
>> com.hp.t4jdbc.HPT4Driver --username narasimharoa@hp.com -P --verbose
>> --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and CNTCT_ID
>> in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
>> --target-dir /home/narasimharao/dataset1
>>
>> successfully able to import the dataset1 and also the .class and .jar
>> files were written to the following location
>>
>> */tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2*
>>
>>
>> *--->below is my command executed to get the second data set after
>> updating the few records which exists in the dataset1 with the same
>> merge-key*
>>
>>
>> sqoop import --connect jdbc:hpt4jdbc://
>> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
>> com.hp.t4jdbc.HPT4Driver --username narasimharoa.chelikani@hp.com -P
>> --verbose --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and
>> CNTCT_ID in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
>> --target-dir /home/narasimharao/dataset2
>>
>> successfully able to import the dataset1 and also the .class and .jar
>> files were written to the following location
>>
>> *
>> /tmp/sqoop-narasimharao/compile/e26282f14a672e7722352fb424781e5a/CNTCT_D.jar
>> *
>>
>>
>> *--->Now the actually Sqoop merge command was given like below:*
>>
>> *sqoop merge --new-data /home/narasimharao/dataset1 --onto
>> /home/narasimharao/dataset2 --target-dir /home/narasimharao/merged
>> --jar-file CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID*
>>
>>
>> Can you please help me which class name i need to mention over here under
>> the class-name attribute and what's wrong in this statement.
>>
>> I tried the below options
>>
>> 1.copied the dataset1 corresponding class and jar files into my home
>> direcotry and tried like below...but getting the same error as below
>>
>> sqoop merge --new-data /home/narasimharao/dataset2 --onto
>> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
>> --jar-file ./CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID
>>
>> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
>> Please set $HBASE_HOME to the root of your HBase installation.
>>
>> 12/02/16 13:56:08 INFO input.FileInputFormat: Total input paths to
>> process : 2
>> 12/02/16 13:56:08 WARN snappy.LoadSnappy: Snappy native library is
>> available
>> 12/02/16 13:56:08 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 12/02/16 13:56:08 INFO snappy.LoadSnappy: Snappy native library loaded
>> 12/02/16 13:56:10 INFO mapred.JobClient: Running job:
>> job_201202161202_0005
>> 12/02/16 13:56:11 INFO mapred.JobClient: map 0% reduce 0%
>> 12/02/16 13:56:55 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_0, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:56:59 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_0, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:22 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_1, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:23 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_1, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:50 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000000_2, Status : FAILED
>> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> * at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> 12/02/16 13:57:51 INFO mapred.JobClient: Task Id :
>> attempt_201202161202_0005_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> CNTCT_D.class
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLo
>> attempt_201202161202_0005_m_000001_2: log4j:WARN No appenders could be
>> found for logger (org.apache.hadoop.mapred.Task).
>> attempt_201202161202_0005_m_000001_2: log4j:WARN Please initialize the
>> log4j system properly.
>> 12/02/16 13:58:27 INFO mapred.JobClient: Job complete:
>> job_201202161202_0005
>> 12/02/16 13:58:28 INFO mapred.JobClient: Counters: 7
>> 12/02/16 13:58:28 INFO mapred.JobClient: Job Counters
>> 12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=249623
>> 12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 12/02/16 13:58:28 INFO mapred.JobClient: Launched map tasks=8
>> 12/02/16 13:58:28 INFO mapred.JobClient: Data-local map tasks=8
>> 12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> 12/02/16 13:58:28 INFO mapred.JobClient: Failed map tasks=1
>> 12/02/16 13:58:28 ERROR tool.MergeTool: MapReduce job failed!
>>
>> 2. given second import dataset2 corresponding classname..but the same
>> error it is also throwing
>>
>> 3.mentioned with full path for both .class and .jar file
>>
>> sqoop merge --new-data /home/narasimharao/dataset2 --onto
>> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
>> --jar-file
>> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.jar
>> --class-name
>> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.class
>> --merge-key CNTCT_ID
>> 4.tried the above stmt3 with second dataset .class and .jar files
>> also..getting the same errors
>>
>>
>>
>> Any Help Appreciated!!!!!!
>>
>> Thanks,
>> Narasimharao
>>
>>
>>
>
>
Re: need help on merge tool in sqoop
Posted by chelikani narasimharao <na...@gmail.com>.
hi guys,
Can any one please help me for the below issue.
Advanced Thanks,
Narasimharao
On Thu, Feb 16, 2012 at 8:24 PM, chelikani narasimharao <
narasimharaoc@gmail.com> wrote:
> Hi All,
>
> I am trying to merge the two datasets using the merge tool
> functionality in sqoop.But i got struck up with the following error.
>
> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
> CNTCT_D.class*
>
> *---->below is my command executed to get the first data set specifying
> the location to store the first dataset to dataset1 under narasimharao
> directory*
>
> sqoop import --connect jdbc:hpt4jdbc://
> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
> com.hp.t4jdbc.HPT4Driver --username narasimharoa@hp.com -P --verbose
> --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and CNTCT_ID
> in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
> --target-dir /home/narasimharao/dataset1
>
> successfully able to import the dataset1 and also the .class and .jar
> files were written to the following location
>
> */tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2*
>
>
> *--->below is my command executed to get the second data set after
> updating the few records which exists in the dataset1 with the same
> merge-key*
>
>
> sqoop import --connect jdbc:hpt4jdbc://
> g4n0401a.houston.hp.com:18650/:schema=EDW_DEV --driver
> com.hp.t4jdbc.HPT4Driver --username narasimharoa.chelikani@hp.com -P
> --verbose --split-by SRC_SYS_KY --table CNTCT_D --where "SRC_SYS_KY=363 and
> CNTCT_ID in('100838','101324','101474','102212','100654') and ACT_FG='Y'"
> --target-dir /home/narasimharao/dataset2
>
> successfully able to import the dataset1 and also the .class and .jar
> files were written to the following location
>
> *
> /tmp/sqoop-narasimharao/compile/e26282f14a672e7722352fb424781e5a/CNTCT_D.jar
> *
>
>
> *--->Now the actually Sqoop merge command was given like below:*
>
> *sqoop merge --new-data /home/narasimharao/dataset1 --onto
> /home/narasimharao/dataset2 --target-dir /home/narasimharao/merged
> --jar-file CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID*
>
>
> Can you please help me which class name i need to mention over here under
> the class-name attribute and what's wrong in this statement.
>
> I tried the below options
>
> 1.copied the dataset1 corresponding class and jar files into my home
> direcotry and tried like below...but getting the same error as below
>
> sqoop merge --new-data /home/narasimharao/dataset2 --onto
> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
> --jar-file ./CNTCT_D.jar --class-name CNTCT_D.class --merge-key CNTCT_ID
>
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
>
> 12/02/16 13:56:08 INFO input.FileInputFormat: Total input paths to process
> : 2
> 12/02/16 13:56:08 WARN snappy.LoadSnappy: Snappy native library is
> available
> 12/02/16 13:56:08 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 12/02/16 13:56:08 INFO snappy.LoadSnappy: Snappy native library loaded
> 12/02/16 13:56:10 INFO mapred.JobClient: Running job: job_201202161202_0005
> 12/02/16 13:56:11 INFO mapred.JobClient: map 0% reduce 0%
> 12/02/16 13:56:55 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000000_0, Status : FAILED
> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
> CNTCT_D.class
> * at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> 12/02/16 13:56:59 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000001_0, Status : FAILED
> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
> CNTCT_D.class
> * at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> 12/02/16 13:57:22 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000000_1, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> 12/02/16 13:57:23 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000001_1, Status : FAILED
> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
> CNTCT_D.class
> * at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> 12/02/16 13:57:50 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000000_2, Status : FAILED
> *java.lang.RuntimeException: java.lang.ClassNotFoundException:
> CNTCT_D.class
> * at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> 12/02/16 13:57:51 INFO mapred.JobClient: Task Id :
> attempt_201202161202_0005_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException: CNTCT_D.class
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> com.cloudera.sqoop.mapreduce.MergeTextMapper.setup(MergeTextMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException: CNTCT_D.class
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLo
> attempt_201202161202_0005_m_000001_2: log4j:WARN No appenders could be
> found for logger (org.apache.hadoop.mapred.Task).
> attempt_201202161202_0005_m_000001_2: log4j:WARN Please initialize the
> log4j system properly.
> 12/02/16 13:58:27 INFO mapred.JobClient: Job complete:
> job_201202161202_0005
> 12/02/16 13:58:28 INFO mapred.JobClient: Counters: 7
> 12/02/16 13:58:28 INFO mapred.JobClient: Job Counters
> 12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=249623
> 12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 12/02/16 13:58:28 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 12/02/16 13:58:28 INFO mapred.JobClient: Launched map tasks=8
> 12/02/16 13:58:28 INFO mapred.JobClient: Data-local map tasks=8
> 12/02/16 13:58:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
> 12/02/16 13:58:28 INFO mapred.JobClient: Failed map tasks=1
> 12/02/16 13:58:28 ERROR tool.MergeTool: MapReduce job failed!
>
> 2. given second import dataset2 corresponding classname..but the same
> error it is also throwing
>
> 3.mentioned with full path for both .class and .jar file
>
> sqoop merge --new-data /home/narasimharao/dataset2 --onto
> /home/narasimharao/dataset1 --target-dir /home/narasimharao/merged
> --jar-file
> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.jar
> --class-name
> /tmp/sqoop-narasimharao/compile/08c0cc665bd3e89124987a7c297827a2/CNTCT_D.class
> --merge-key CNTCT_ID
> 4.tried the above stmt3 with second dataset .class and .jar files
> also..getting the same errors
>
>
>
> Any Help Appreciated!!!!!!
>
> Thanks,
> Narasimharao
>
>
>