You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Jane Wayne <ja...@gmail.com> on 2013/06/06 02:23:09 UTC
can we use -libjars with the Sqoop tool?
hi,
i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
tools from the command line using the -libjars flag/option/parameter?
i have tried the following.
export
LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop
what i get is the message below.
No such sqoop tool: -libjars. See 'sqoop help'.
obviously, this problem has something to do with sqoop's parsing of the
command line options. shouldn't sqoop allow hadoop's command line options
and its own at the same time?
in the meanwhile, i will try to tinker a bit with that Sqoop class to see
if i can get it to work with -libjars.
references that i found regarding -libjars.
1. http://grepalex.com/2013/02/25/hadoop-libjars/
2.
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
thanks,
Re: can we use -libjars with the Sqoop tool?
Posted by abhijeet gaikwad <ab...@gmail.com>.
With the approach you are taking, see if this helps:
export HADOOP_CLASSPATH=$LIBJARS
-libjars makes sure each MR task gets these jars in classpath but for the
local machine that kick starts hadoop job does not get these jars, so the
CNFE. You need to explicitly do that as above.
A better solution:
If your edge node can submit a hadoop job to the cluster, I think this
should be easy: Install Sqoop on Edge node and run the command that Venkat
suggested.
Let me know how it goes!
Thanks,
Abhijeet
On Fri, Jun 7, 2013 at 1:41 AM, Jane Wayne <ja...@gmail.com> wrote:
> venkat,
>
> could you clarify? do you mean to not use the hadoop jar command (option
> 1)? or to use the hadoop jar command and omit -config (option 2)?
>
> option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
> sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
> ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
> mypass --table data --verbose --append --warehouse-dir /sqoop
>
> i need -libjars because i am trying to run sqoop from an edge node that is
> not part of the hadoop cluster. is this even possible?
>
> i posted this question on the dev mailing list, since i thought i might
> have to modify sqoop. but what i am trying to explore is this scenario.
>
> W: web server (JEE container, tomcat)
> H: hadoop cluster
> D: database server (MySQL)
>
> i need to invoke sqoop on W to import data from D to H. i read that sqoop2
> has a client API, which should have met my requirements. however, i am not
> allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
> cluster).
>
> i am opened to any approach subject to that one constraint (cannot install
> sqoop onto the hadoop cluster or put third-party libs onto the hadoop
> cluster).
>
> thanks.
>
>
>
> On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> Don't use hadoop -config ...
>>
>> Use
>> sqoop import -libjars .... And I am not sure why you need libjars
>> option. For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
>> and just run sqoop import
>>
>>
>> Venkat
>>
>>
>> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <ja...@gmail.com>wrote:
>>
>>> i am running the Sqoop tool as follows.
>>>
>>> export
>>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> i am getting this stack trace.
>>>
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>> at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>> at
>>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>>> at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>
>>> when i looked at the sqoop-1.4.3.jar, clearly,
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>>>
>>> any ideas what's going on?
>>>
>>>
>>>
>>>
>>> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
>>> vranganathan@hortonworks.com> wrote:
>>>
>>>> sqoop tools implement the ToolRunner interface. So, you can pass
>>>> -libjars to sqoop (but make sure it is the first option among the list of
>>>> the options
>>>>
>>>> For example, sqoop import -libjars $LIBJARS other options
>>>>
>>>> Venkat
>>>>
>>>>
>>>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com>wrote:
>>>>
>>>>> hi,
>>>>>
>>>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>>>> tools from the command line using the -libjars flag/option/parameter?
>>>>>
>>>>> i have tried the following.
>>>>>
>>>>> export
>>>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>>>
>>>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>>>> --verbose --append --warehouse-dir /sqoop
>>>>>
>>>>> what i get is the message below.
>>>>>
>>>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>>>
>>>>> obviously, this problem has something to do with sqoop's parsing of
>>>>> the command line options. shouldn't sqoop allow hadoop's command line
>>>>> options and its own at the same time?
>>>>>
>>>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>>>> see if i can get it to work with -libjars.
>>>>>
>>>>> references that i found regarding -libjars.
>>>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>>>> 2.
>>>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>>>>
>>>>> thanks,
>>>>>
>>>>
>>>>
>>>
>>
>
Re: can we use -libjars with the Sqoop tool?
Posted by Jane Wayne <ja...@gmail.com>.
venkat,
could you clarify? do you mean to not use the hadoop jar command (option
1)? or to use the hadoop jar command and omit -config (option 2)?
option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop
option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
mypass --table data --verbose --append --warehouse-dir /sqoop
i need -libjars because i am trying to run sqoop from an edge node that is
not part of the hadoop cluster. is this even possible?
i posted this question on the dev mailing list, since i thought i might
have to modify sqoop. but what i am trying to explore is this scenario.
W: web server (JEE container, tomcat)
H: hadoop cluster
D: database server (MySQL)
i need to invoke sqoop on W to import data from D to H. i read that sqoop2
has a client API, which should have met my requirements. however, i am not
allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
cluster).
i am opened to any approach subject to that one constraint (cannot install
sqoop onto the hadoop cluster or put third-party libs onto the hadoop
cluster).
thanks.
On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> Don't use hadoop -config ...
>
> Use
> sqoop import -libjars .... And I am not sure why you need libjars
> option. For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
> and just run sqoop import
>
>
> Venkat
>
>
> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <ja...@gmail.com>wrote:
>
>> i am running the Sqoop tool as follows.
>>
>> export
>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>> --verbose --append --warehouse-dir /sqoop
>>
>> i am getting this stack trace.
>>
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>> at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>
>> when i looked at the sqoop-1.4.3.jar, clearly,
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>>
>> any ideas what's going on?
>>
>>
>>
>>
>> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
>> vranganathan@hortonworks.com> wrote:
>>
>>> sqoop tools implement the ToolRunner interface. So, you can pass
>>> -libjars to sqoop (but make sure it is the first option among the list of
>>> the options
>>>
>>> For example, sqoop import -libjars $LIBJARS other options
>>>
>>> Venkat
>>>
>>>
>>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com>wrote:
>>>
>>>> hi,
>>>>
>>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>>> tools from the command line using the -libjars flag/option/parameter?
>>>>
>>>> i have tried the following.
>>>>
>>>> export
>>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>>
>>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>>> --verbose --append --warehouse-dir /sqoop
>>>>
>>>> what i get is the message below.
>>>>
>>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>>
>>>> obviously, this problem has something to do with sqoop's parsing of the
>>>> command line options. shouldn't sqoop allow hadoop's command line options
>>>> and its own at the same time?
>>>>
>>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>>> see if i can get it to work with -libjars.
>>>>
>>>> references that i found regarding -libjars.
>>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>>> 2.
>>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>>>
>>>> thanks,
>>>>
>>>
>>>
>>
>
Re: can we use -libjars with the Sqoop tool?
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
Don't use hadoop -config ...
Use
sqoop import -libjars .... And I am not sure why you need libjars
option. For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
and just run sqoop import
Venkat
On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <ja...@gmail.com> wrote:
> i am running the Sqoop tool as follows.
>
> export
> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>
> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> i am getting this stack trace.
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
> at
> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>
> when i looked at the sqoop-1.4.3.jar, clearly,
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>
> any ideas what's going on?
>
>
>
>
> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> sqoop tools implement the ToolRunner interface. So, you can pass
>> -libjars to sqoop (but make sure it is the first option among the list of
>> the options
>>
>> For example, sqoop import -libjars $LIBJARS other options
>>
>> Venkat
>>
>>
>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com>wrote:
>>
>>> hi,
>>>
>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>> tools from the command line using the -libjars flag/option/parameter?
>>>
>>> i have tried the following.
>>>
>>> export
>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> what i get is the message below.
>>>
>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>
>>> obviously, this problem has something to do with sqoop's parsing of the
>>> command line options. shouldn't sqoop allow hadoop's command line options
>>> and its own at the same time?
>>>
>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>> see if i can get it to work with -libjars.
>>>
>>> references that i found regarding -libjars.
>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>> 2.
>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>>
>>> thanks,
>>>
>>
>>
>
Re: can we use -libjars with the Sqoop tool?
Posted by Deepak Vohra <dv...@yahoo.com>.
can we use -libjars with the Sqoop tool?
Yes.
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
________________________________
From: Jane Wayne <ja...@gmail.com>
To: user@sqoop.apache.org
Sent: Thursday, June 6, 2013 5:56 AM
Subject: Re: can we use -libjars with the Sqoop tool?
i am running the Sqoop tool as follows.
export LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password mypass --table data --verbose --append --warehouse-dir /sqoop
i am getting this stack trace.
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
when i looked at the sqoop-1.4.3.jar, clearly, org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
any ideas what's going on?
On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <vr...@hortonworks.com> wrote:
sqoop tools implement the ToolRunner interface. So, you can pass -libjars to sqoop (but make sure it is the first option among the list of the options
>
>
>For example, sqoop import -libjars $LIBJARS other options
>
>Venkat
>
>
>
>On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com> wrote:
>
>hi,
>>
>>
>>i am wondering if i can run the org.apache.sqoop.Sqoop tool like other tools from the command line using the -libjars flag/option/parameter?
>>
>>
>>i have tried the following.
>>
>>
>>export LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>>
>>hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect jdbc:mysql://mysql/mydb --username myuser --password mypass --table data --verbose --append --warehouse-dir /sqoop
>>
>>
>>what i get is the message below.
>>
>>
>>No such sqoop tool: -libjars. See 'sqoop help'.
>>
>>
>>
>>obviously, this problem has something to do with sqoop's parsing of the command line options. shouldn't sqoop allow hadoop's command line options and its own at the same time?
>>
>>
>>in the meanwhile, i will try to tinker a bit with that Sqoop class to see if i can get it to work with -libjars.
>>
>>
>>references that i found regarding -libjars.
>>1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>2. http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>
>>
>>thanks,
>
Re: can we use -libjars with the Sqoop tool?
Posted by Jane Wayne <ja...@gmail.com>.
i am running the Sqoop tool as follows.
export
LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop
i am getting this stack trace.
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
at
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException:
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
when i looked at the sqoop-1.4.3.jar, clearly,
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
any ideas what's going on?
On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> sqoop tools implement the ToolRunner interface. So, you can pass
> -libjars to sqoop (but make sure it is the first option among the list of
> the options
>
> For example, sqoop import -libjars $LIBJARS other options
>
> Venkat
>
>
> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com>wrote:
>
>> hi,
>>
>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>> tools from the command line using the -libjars flag/option/parameter?
>>
>> i have tried the following.
>>
>> export
>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>> --verbose --append --warehouse-dir /sqoop
>>
>> what i get is the message below.
>>
>> No such sqoop tool: -libjars. See 'sqoop help'.
>>
>> obviously, this problem has something to do with sqoop's parsing of the
>> command line options. shouldn't sqoop allow hadoop's command line options
>> and its own at the same time?
>>
>> in the meanwhile, i will try to tinker a bit with that Sqoop class to see
>> if i can get it to work with -libjars.
>>
>> references that i found regarding -libjars.
>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>> 2.
>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>
>> thanks,
>>
>
>
Re: can we use -libjars with the Sqoop tool?
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
sqoop tools implement the ToolRunner interface. So, you can pass
-libjars to sqoop (but make sure it is the first option among the list of
the options
For example, sqoop import -libjars $LIBJARS other options
Venkat
On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <ja...@gmail.com> wrote:
> hi,
>
> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
> tools from the command line using the -libjars flag/option/parameter?
>
> i have tried the following.
>
> export
> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>
> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> what i get is the message below.
>
> No such sqoop tool: -libjars. See 'sqoop help'.
>
> obviously, this problem has something to do with sqoop's parsing of the
> command line options. shouldn't sqoop allow hadoop's command line options
> and its own at the same time?
>
> in the meanwhile, i will try to tinker a bit with that Sqoop class to see
> if i can get it to work with -libjars.
>
> references that i found regarding -libjars.
> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
> 2.
> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>
> thanks,
>