You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Sebastian Gäde <s1...@hft-leipzig.de> on 2014/06/13 13:15:50 UTC
using Sqoop 1.4.4 with Hadoop 2.4.0
Hi,
Id like to export MR result (text) files from HDFS to MySQL. My cluster is
running Hadoop 2.4.0.
When Im running
./sqoop export connect jdbc:mysql://mysqlserver/sqoop username user
password pass table test1 export-dir /output
I get the below exception. I assume, sqoop is using the old MR API? Can I
change this somehow to use the new API, then I'd be able to point
$HADOOP_MAPRED_HOME to the right position?!
14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
streaming resultset.
14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `test1` AS t LIMIT 1
14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `test1` AS t LIMIT 1
14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/opt/hadoop-2.4.0/share/hadoop/yarn
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/JobConf
at
org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager
.java:109)
at
org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
at
org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Thanks very much!
Sebastian
AW: using Sqoop 1.4.4 with Hadoop 2.4.0
Posted by Gäde,
Sebastian <s1...@hft-leipzig.de>.
Thanks Abe,
This is the output:
<snip>
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-flume-1.4.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/16 08:54:19 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/06/16 08:54:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/06/16 08:54:19 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql:
14/06/16 08:54:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@1537e1d
14/06/16 08:54:19 INFO tool.CodeGenTool: Beginning code generation
14/06/16 08:54:19 DEBUG manager.SqlManager: Execute getColumnTypesRawQuery : SELECT t.* FROM `test1` AS t LIMIT 1
14/06/16 08:54:19 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1
14/06/16 08:54:24 DEBUG orm.ClassWriter: selected columns:
14/06/16 08:54:24 DEBUG orm.ClassWriter: index
14/06/16 08:54:24 DEBUG orm.ClassWriter: bel
14/06/16 08:54:24 DEBUG orm.ClassWriter: gesp
14/06/16 08:54:24 DEBUG orm.ClassWriter: bd
14/06/16 08:54:24 DEBUG orm.ClassWriter: gd1
14/06/16 08:54:24 DEBUG orm.ClassWriter: gd2
14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1
14/06/16 08:54:24 DEBUG orm.ClassWriter: Writing source file: /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/test1.java
14/06/16 08:54:24 DEBUG orm.ClassWriter: Table name: test1
14/06/16 08:54:24 DEBUG orm.ClassWriter: Columns: index:12, bel:4, gesp:4, bd:4, gd1:4, gd2:4,
14/06/16 08:54:24 DEBUG orm.ClassWriter: sourceFilename is test1.java
14/06/16 08:54:24 DEBUG orm.CompilationManager: Found existing /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/
14/06/16 08:54:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.4.0/share/hadoop/mapreduce
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
at org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager.java:109)
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 11 more
</snip>
In the past such issues turned out to be caused by a wrong classpath. However, I assume Sqoop is building the classpath on its own depending on the input for HADOOP_*_HOME?!
There are no MR logs.
Cheers
Seb.
________________________________
Von: Abraham Elmahrek [abe@cloudera.com]
Gesendet: Sonntag, 15. Juni 2014 23:54
An: user@sqoop.apache.org
Betreff: Re: using Sqoop 1.4.4 with Hadoop 2.4.0
You can add --verbose to the beginning of your command (right after the word "export") and you'll get some more info. Could you copy/paste the output of your sqoop command with this flag? There should also be logs in the MapReduce job/task logs. In this case, I think the application and container logs may have more information. They are usually located under '/var/log/hadoop-yarn/'. Have you considered installing these components using a vendor package or BigTop?
On Sun, Jun 15, 2014 at 12:42 AM, Sebastian Gäde <s1...@hft-leipzig.de>> wrote:
Hi,
Thanks Abe.
Yes, Sqoop and Hadoop were downloaded from a tarball.
I set HADOOP_COMMON_HOME to /opt/hadoop-2.4.0 and HADOOP_MAPRED_HOME to /opt/hadoop-2.4.0/share/hadoop/yarn using the conf/sqoop-env.sh. I also tried /opt/hadoop-2.4.0/share/hadoop/mapreduce for HADOOP_MAPRED_HOME as I did not know which JAR I need. However, the error was the same.
Is there a way to enable DEBUG messages? This sometimes helped me in the passed with other Hadoop issues.
Cheers
Seb.
Am 14.06.2014 06:37, schrieb Abraham Elmahrek:
Hey there,
The exception is indicative of a missing jar from the class path. It
seems like the hadoop-mapreduce-client-core.jar file is missing. Have
you set HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME?
How did you install Sqoop and Hadoop? If you've downloaded a tarball,
you may need to set the environment variables specified above.
For more info see
http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_controlling_the_hadoop_installation.
-Abe
On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <s1...@hft-leipzig.de>
<ma...@hft-leipzig.de>>> wrote:
Hi,
I’d like to export MR result (text) files from HDFS to MySQL. My
cluster is
running Hadoop 2.4.0.
When I’m running
./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user
–password pass –table test1 –export-dir /output
I get the below exception. I assume, sqoop is using the old MR API?
Can I
change this somehow to use the new API, then I'd be able to point
$HADOOP_MAPRED_HOME to the right position?!
14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
streaming resultset.
14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
SELECT
t.* FROM `test1` AS t LIMIT 1
14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
SELECT
t.* FROM `test1` AS t LIMIT 1
14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/opt/hadoop-2.4.0/share/hadoop/yarn
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/JobConf
at
org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager
.java:109)
at
org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
at
org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at
org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Thanks very much!
Sebastian
Re: using Sqoop 1.4.4 with Hadoop 2.4.0
Posted by Abraham Elmahrek <ab...@cloudera.com>.
You can add --verbose to the beginning of your command (right after the
word "export") and you'll get some more info. Could you copy/paste the
output of your sqoop command with this flag? There should also be logs in
the MapReduce job/task logs. In this case, I think the application and
container logs may have more information. They are usually located under
'/var/log/hadoop-yarn/'. Have you considered installing these components
using a vendor package or BigTop?
On Sun, Jun 15, 2014 at 12:42 AM, Sebastian Gäde <s1...@hft-leipzig.de>
wrote:
> Hi,
>
> Thanks Abe.
>
> Yes, Sqoop and Hadoop were downloaded from a tarball.
>
> I set HADOOP_COMMON_HOME to /opt/hadoop-2.4.0 and HADOOP_MAPRED_HOME to
> /opt/hadoop-2.4.0/share/hadoop/yarn using the conf/sqoop-env.sh. I also
> tried /opt/hadoop-2.4.0/share/hadoop/mapreduce for HADOOP_MAPRED_HOME as
> I did not know which JAR I need. However, the error was the same.
>
> Is there a way to enable DEBUG messages? This sometimes helped me in the
> passed with other Hadoop issues.
>
> Cheers
> Seb.
>
> Am 14.06.2014 06:37, schrieb Abraham Elmahrek:
>
>> Hey there,
>>
>> The exception is indicative of a missing jar from the class path. It
>> seems like the hadoop-mapreduce-client-core.jar file is missing. Have
>> you set HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME?
>>
>> How did you install Sqoop and Hadoop? If you've downloaded a tarball,
>> you may need to set the environment variables specified above.
>>
>> For more info see
>> http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_
>> controlling_the_hadoop_installation.
>>
>> -Abe
>>
>>
>> On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <s116102@hft-leipzig.de
>> <ma...@hft-leipzig.de>> wrote:
>>
>> Hi,
>>
>> I’d like to export MR result (text) files from HDFS to MySQL. My
>> cluster is
>> running Hadoop 2.4.0.
>>
>> When I’m running
>> ./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user
>> –password pass –table test1 –export-dir /output
>> I get the below exception. I assume, sqoop is using the old MR API?
>> Can I
>> change this somehow to use the new API, then I'd be able to point
>> $HADOOP_MAPRED_HOME to the right position?!
>>
>> 14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on
>> the
>> command-line is insecure. Consider using -P instead.
>> 14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
>> streaming resultset.
>> 14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
>> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
>> SELECT
>> t.* FROM `test1` AS t LIMIT 1
>> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
>> SELECT
>> t.* FROM `test1` AS t LIMIT 1
>> 14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>> /opt/hadoop-2.4.0/share/hadoop/yarn
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapred/JobConf
>> at
>> org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(
>> CompilationManager
>> .java:109)
>> at
>> org.apache.sqoop.orm.CompilationManager.compile(
>> CompilationManager.java:137)
>> at
>> org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
>> at
>> org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
>> at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
>> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>>
>> Thanks very much!
>> Sebastian
>>
>>
>>
>>
Re: using Sqoop 1.4.4 with Hadoop 2.4.0
Posted by Sebastian Gäde <s1...@hft-leipzig.de>.
Hi,
Thanks Abe.
Yes, Sqoop and Hadoop were downloaded from a tarball.
I set HADOOP_COMMON_HOME to /opt/hadoop-2.4.0 and HADOOP_MAPRED_HOME to
/opt/hadoop-2.4.0/share/hadoop/yarn using the conf/sqoop-env.sh. I also
tried /opt/hadoop-2.4.0/share/hadoop/mapreduce for HADOOP_MAPRED_HOME as
I did not know which JAR I need. However, the error was the same.
Is there a way to enable DEBUG messages? This sometimes helped me in the
passed with other Hadoop issues.
Cheers
Seb.
Am 14.06.2014 06:37, schrieb Abraham Elmahrek:
> Hey there,
>
> The exception is indicative of a missing jar from the class path. It
> seems like the hadoop-mapreduce-client-core.jar file is missing. Have
> you set HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME?
>
> How did you install Sqoop and Hadoop? If you've downloaded a tarball,
> you may need to set the environment variables specified above.
>
> For more info see
> http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_controlling_the_hadoop_installation.
>
> -Abe
>
>
> On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <s116102@hft-leipzig.de
> <ma...@hft-leipzig.de>> wrote:
>
> Hi,
>
> I’d like to export MR result (text) files from HDFS to MySQL. My
> cluster is
> running Hadoop 2.4.0.
>
> When I’m running
> ./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user
> –password pass –table test1 –export-dir /output
> I get the below exception. I assume, sqoop is using the old MR API?
> Can I
> change this somehow to use the new API, then I'd be able to point
> $HADOOP_MAPRED_HOME to the right position?!
>
> 14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
> streaming resultset.
> 14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
> SELECT
> t.* FROM `test1` AS t LIMIT 1
> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement:
> SELECT
> t.* FROM `test1` AS t LIMIT 1
> 14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /opt/hadoop-2.4.0/share/hadoop/yarn
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/JobConf
> at
> org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager
> .java:109)
> at
> org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
> at
> org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> at
> org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
> at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>
> Thanks very much!
> Sebastian
>
>
>
Re: using Sqoop 1.4.4 with Hadoop 2.4.0
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey there,
The exception is indicative of a missing jar from the class path. It seems
like the hadoop-mapreduce-client-core.jar file is missing. Have you set
HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME?
How did you install Sqoop and Hadoop? If you've downloaded a tarball, you
may need to set the environment variables specified above.
For more info see
http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_controlling_the_hadoop_installation
.
-Abe
On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <s1...@hft-leipzig.de>
wrote:
> Hi,
>
> I’d like to export MR result (text) files from HDFS to MySQL. My cluster is
> running Hadoop 2.4.0.
>
> When I’m running
> ./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user
> –password pass –table test1 –export-dir /output
> I get the below exception. I assume, sqoop is using the old MR API? Can I
> change this somehow to use the new API, then I'd be able to point
> $HADOOP_MAPRED_HOME to the right position?!
>
> 14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL
> streaming resultset.
> 14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation
> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `test1` AS t LIMIT 1
> 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `test1` AS t LIMIT 1
> 14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /opt/hadoop-2.4.0/share/hadoop/yarn
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/JobConf
> at
>
> org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager
> .java:109)
> at
>
> org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137)
> at
> org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
> at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>
> Thanks very much!
> Sebastian
>
>
>