You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Justin Lee <ju...@mongodb.com> on 2014/07/18 13:34:15 UTC
sqoop2 submission problems.
I'm submitting a job programmatically as detailed here (
http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
Exception info : java.io.FileNotFoundException: File does not exist: hdfs://
127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
I've seen some jobs (hive, iirc) fail and I've had to copy jars on to hdfs
to resolve it but I'm not sure why I'd need to do this with sqoop jars. If
this is necessary for sqoop2 to function, then sqoop2 should take care of
this step. Am I missing some step not mentioned on that page?
*{ *name : *"**Justin Lee**", *
title : *"*Software Engineer*"*,
twitter : *"*@evanchooly <http://twitter.com/evanchooly>*"*,
web : *[* "10gen.com", *"*antwerkz.com*"* *]*,
location : *"*New York, NY*"* *}*
Re: sqoop2 submission problems.
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
I would recommend to check configuration files that you have correct configs and libraries. This exception is very common when configs does not correspond to the libraries that are used.
Did you followed up the installation docs at:
http://sqoop.apache.org/docs/1.99.3/Installation.html
Jarcec
On Jul 23, 2014, at 11:32 PM, Justin Lee <ju...@mongodb.com> wrote:
> It's a stock sqoop 1.99.3 and hadoop 2.4 download.
>
> --------------------------------
>
> { name : "Justin Lee",
> title : "Software Engineer",
> twitter : "@evanchooly",
> web : [ "10gen.com", "antwerkz.com" ],
> location : "New York, NY" }
>
>
> On Wed, Jul 23, 2014 at 10:39 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> What about distribution? Tarballs downloaded from Apache website or any commercial distribution?
>
> Jarcec
>
> On Jul 22, 2014, at 7:40 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > 2.4
> >
> > --------------------------------
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
> >
> >
> > On Tue, Jul 22, 2014 at 4:32 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> > What Hadoop distribution and version are you using?
> >
> > Jarcec
> >
> > On Jul 22, 2014, at 7:29 AM, Justin Lee <ju...@mongodb.com> wrote:
> >
> > > I got sidetracked by other issues but I'm still having this problem. I'm not sure what's missing here. Do I need to copy any sqoop jars over to the hadoop lib folders? I'd be less confused if it was my own stuff it couldn't find. I followed the directions (http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem to have missed anything. Any ideas?
> > >
> > > --------------------------------
> > >
> > > { name : "Justin Lee",
> > > title : "Software Engineer",
> > > twitter : "@evanchooly",
> > > web : [ "10gen.com", "antwerkz.com" ],
> > > location : "New York, NY" }
> > >
> > >
> > > On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> > > I’ve seen this particular problem several times. So far it was always caused by Sqoop 2 server miss configuration in map reduce libraries and configuration - either by having configured yarn (MR2) while having MR1 libraries on the class path or vice versa. Hence I would suggest to verify the deployment.
> > >
> > > Jarcec
> > >
> > > On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
> > >
> > > > I'm submitting a job programmatically as detailed here (http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> > > >
> > > > Exception info : java.io.FileNotFoundException: File does not exist: hdfs://127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> > > >
> > > > I've seen some jobs (hive, iirc) fail and I've had to copy jars on to hdfs to resolve it but I'm not sure why I'd need to do this with sqoop jars. If this is necessary for sqoop2 to function, then sqoop2 should take care of this step. Am I missing some step not mentioned on that page?
> > > >
> > > >
> > > > { name : "Justin Lee",
> > > > title : "Software Engineer",
> > > > twitter : "@evanchooly",
> > > > web : [ "10gen.com", "antwerkz.com" ],
> > > > location : "New York, NY" }
> > >
> > >
> >
> >
>
>
Re: sqoop2 submission problems.
Posted by Justin Lee <ju...@mongodb.com>.
It's a stock sqoop 1.99.3 and hadoop 2.4 download.
*--------------------------------*
*{ *name : *"**Justin Lee**", *
title : *"*Software Engineer*"*,
twitter : *"*@evanchooly <http://twitter.com/evanchooly>*"*,
web : *[* "10gen.com", *"*antwerkz.com*"* *]*,
location : *"*New York, NY*"* *}*
On Wed, Jul 23, 2014 at 10:39 PM, Jarek Jarcec Cecho <ja...@apache.org>
wrote:
> What about distribution? Tarballs downloaded from Apache website or any
> commercial distribution?
>
> Jarcec
>
> On Jul 22, 2014, at 7:40 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > 2.4
> >
> > --------------------------------
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
> >
> >
> > On Tue, Jul 22, 2014 at 4:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
> wrote:
> > What Hadoop distribution and version are you using?
> >
> > Jarcec
> >
> > On Jul 22, 2014, at 7:29 AM, Justin Lee <ju...@mongodb.com> wrote:
> >
> > > I got sidetracked by other issues but I'm still having this problem.
> I'm not sure what's missing here. Do I need to copy any sqoop jars over
> to the hadoop lib folders? I'd be less confused if it was my own stuff it
> couldn't find. I followed the directions (
> http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem
> to have missed anything. Any ideas?
> > >
> > > --------------------------------
> > >
> > > { name : "Justin Lee",
> > > title : "Software Engineer",
> > > twitter : "@evanchooly",
> > > web : [ "10gen.com", "antwerkz.com" ],
> > > location : "New York, NY" }
> > >
> > >
> > > On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org>
> wrote:
> > > I’ve seen this particular problem several times. So far it was always
> caused by Sqoop 2 server miss configuration in map reduce libraries and
> configuration - either by having configured yarn (MR2) while having MR1
> libraries on the class path or vice versa. Hence I would suggest to verify
> the deployment.
> > >
> > > Jarcec
> > >
> > > On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com>
> wrote:
> > >
> > > > I'm submitting a job programmatically as detailed here (
> http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> > > >
> > > > Exception info : java.io.FileNotFoundException: File does not exist:
> hdfs://
> 127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> > > >
> > > > I've seen some jobs (hive, iirc) fail and I've had to copy jars on
> to hdfs to resolve it but I'm not sure why I'd need to do this with sqoop
> jars. If this is necessary for sqoop2 to function, then sqoop2 should take
> care of this step. Am I missing some step not mentioned on that page?
> > > >
> > > >
> > > > { name : "Justin Lee",
> > > > title : "Software Engineer",
> > > > twitter : "@evanchooly",
> > > > web : [ "10gen.com", "antwerkz.com" ],
> > > > location : "New York, NY" }
> > >
> > >
> >
> >
>
>
Re: sqoop2 submission problems.
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
What about distribution? Tarballs downloaded from Apache website or any commercial distribution?
Jarcec
On Jul 22, 2014, at 7:40 AM, Justin Lee <ju...@mongodb.com> wrote:
> 2.4
>
> --------------------------------
>
> { name : "Justin Lee",
> title : "Software Engineer",
> twitter : "@evanchooly",
> web : [ "10gen.com", "antwerkz.com" ],
> location : "New York, NY" }
>
>
> On Tue, Jul 22, 2014 at 4:32 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> What Hadoop distribution and version are you using?
>
> Jarcec
>
> On Jul 22, 2014, at 7:29 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > I got sidetracked by other issues but I'm still having this problem. I'm not sure what's missing here. Do I need to copy any sqoop jars over to the hadoop lib folders? I'd be less confused if it was my own stuff it couldn't find. I followed the directions (http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem to have missed anything. Any ideas?
> >
> > --------------------------------
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
> >
> >
> > On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> > I’ve seen this particular problem several times. So far it was always caused by Sqoop 2 server miss configuration in map reduce libraries and configuration - either by having configured yarn (MR2) while having MR1 libraries on the class path or vice versa. Hence I would suggest to verify the deployment.
> >
> > Jarcec
> >
> > On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
> >
> > > I'm submitting a job programmatically as detailed here (http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> > >
> > > Exception info : java.io.FileNotFoundException: File does not exist: hdfs://127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> > >
> > > I've seen some jobs (hive, iirc) fail and I've had to copy jars on to hdfs to resolve it but I'm not sure why I'd need to do this with sqoop jars. If this is necessary for sqoop2 to function, then sqoop2 should take care of this step. Am I missing some step not mentioned on that page?
> > >
> > >
> > > { name : "Justin Lee",
> > > title : "Software Engineer",
> > > twitter : "@evanchooly",
> > > web : [ "10gen.com", "antwerkz.com" ],
> > > location : "New York, NY" }
> >
> >
>
>
Re: sqoop2 submission problems.
Posted by Justin Lee <ju...@mongodb.com>.
2.4
*--------------------------------*
*{ *name : *"**Justin Lee**", *
title : *"*Software Engineer*"*,
twitter : *"*@evanchooly <http://twitter.com/evanchooly>*"*,
web : *[* "10gen.com", *"*antwerkz.com*"* *]*,
location : *"*New York, NY*"* *}*
On Tue, Jul 22, 2014 at 4:32 PM, Jarek Jarcec Cecho <ja...@apache.org>
wrote:
> What Hadoop distribution and version are you using?
>
> Jarcec
>
> On Jul 22, 2014, at 7:29 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > I got sidetracked by other issues but I'm still having this problem.
> I'm not sure what's missing here. Do I need to copy any sqoop jars over
> to the hadoop lib folders? I'd be less confused if it was my own stuff it
> couldn't find. I followed the directions (
> http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem
> to have missed anything. Any ideas?
> >
> > --------------------------------
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
> >
> >
> > On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org>
> wrote:
> > I’ve seen this particular problem several times. So far it was always
> caused by Sqoop 2 server miss configuration in map reduce libraries and
> configuration - either by having configured yarn (MR2) while having MR1
> libraries on the class path or vice versa. Hence I would suggest to verify
> the deployment.
> >
> > Jarcec
> >
> > On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
> >
> > > I'm submitting a job programmatically as detailed here (
> http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> > >
> > > Exception info : java.io.FileNotFoundException: File does not exist:
> hdfs://
> 127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> > >
> > > I've seen some jobs (hive, iirc) fail and I've had to copy jars on to
> hdfs to resolve it but I'm not sure why I'd need to do this with sqoop
> jars. If this is necessary for sqoop2 to function, then sqoop2 should take
> care of this step. Am I missing some step not mentioned on that page?
> > >
> > >
> > > { name : "Justin Lee",
> > > title : "Software Engineer",
> > > twitter : "@evanchooly",
> > > web : [ "10gen.com", "antwerkz.com" ],
> > > location : "New York, NY" }
> >
> >
>
>
Re: sqoop2 submission problems.
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
What Hadoop distribution and version are you using?
Jarcec
On Jul 22, 2014, at 7:29 AM, Justin Lee <ju...@mongodb.com> wrote:
> I got sidetracked by other issues but I'm still having this problem. I'm not sure what's missing here. Do I need to copy any sqoop jars over to the hadoop lib folders? I'd be less confused if it was my own stuff it couldn't find. I followed the directions (http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem to have missed anything. Any ideas?
>
> --------------------------------
>
> { name : "Justin Lee",
> title : "Software Engineer",
> twitter : "@evanchooly",
> web : [ "10gen.com", "antwerkz.com" ],
> location : "New York, NY" }
>
>
> On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
> I’ve seen this particular problem several times. So far it was always caused by Sqoop 2 server miss configuration in map reduce libraries and configuration - either by having configured yarn (MR2) while having MR1 libraries on the class path or vice versa. Hence I would suggest to verify the deployment.
>
> Jarcec
>
> On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > I'm submitting a job programmatically as detailed here (http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> >
> > Exception info : java.io.FileNotFoundException: File does not exist: hdfs://127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> >
> > I've seen some jobs (hive, iirc) fail and I've had to copy jars on to hdfs to resolve it but I'm not sure why I'd need to do this with sqoop jars. If this is necessary for sqoop2 to function, then sqoop2 should take care of this step. Am I missing some step not mentioned on that page?
> >
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
>
>
Re: sqoop2 submission problems.
Posted by Justin Lee <ju...@mongodb.com>.
I got sidetracked by other issues but I'm still having this problem. I'm
not sure what's missing here. Do I need to copy any sqoop jars over to the
hadoop lib folders? I'd be less confused if it was my own stuff it
couldn't find. I followed the directions (
http://sqoop.apache.org/docs/1.99.3/Installation.html) and I don't seem to
have missed anything. Any ideas?
*--------------------------------*
*{ *name : *"**Justin Lee**", *
title : *"*Software Engineer*"*,
twitter : *"*@evanchooly <http://twitter.com/evanchooly>*"*,
web : *[* "10gen.com", *"*antwerkz.com*"* *]*,
location : *"*New York, NY*"* *}*
On Fri, Jul 18, 2014 at 3:19 PM, Jarek Jarcec Cecho <ja...@apache.org>
wrote:
> I’ve seen this particular problem several times. So far it was always
> caused by Sqoop 2 server miss configuration in map reduce libraries and
> configuration - either by having configured yarn (MR2) while having MR1
> libraries on the class path or vice versa. Hence I would suggest to verify
> the deployment.
>
> Jarcec
>
> On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
>
> > I'm submitting a job programmatically as detailed here (
> http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
> >
> > Exception info : java.io.FileNotFoundException: File does not exist:
> hdfs://
> 127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
> >
> > I've seen some jobs (hive, iirc) fail and I've had to copy jars on to
> hdfs to resolve it but I'm not sure why I'd need to do this with sqoop
> jars. If this is necessary for sqoop2 to function, then sqoop2 should take
> care of this step. Am I missing some step not mentioned on that page?
> >
> >
> > { name : "Justin Lee",
> > title : "Software Engineer",
> > twitter : "@evanchooly",
> > web : [ "10gen.com", "antwerkz.com" ],
> > location : "New York, NY" }
>
>
Re: sqoop2 submission problems.
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
I’ve seen this particular problem several times. So far it was always caused by Sqoop 2 server miss configuration in map reduce libraries and configuration - either by having configured yarn (MR2) while having MR1 libraries on the class path or vice versa. Hence I would suggest to verify the deployment.
Jarcec
On Jul 18, 2014, at 4:34 AM, Justin Lee <ju...@mongodb.com> wrote:
> I'm submitting a job programmatically as detailed here (http://sqoop.apache.org/docs/1.99.3/ClientAPI.html) and getting an error:
>
> Exception info : java.io.FileNotFoundException: File does not exist: hdfs://127.0.0.1:8020/Users/jlee/hadoop-binaries/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.3.jar
>
> I've seen some jobs (hive, iirc) fail and I've had to copy jars on to hdfs to resolve it but I'm not sure why I'd need to do this with sqoop jars. If this is necessary for sqoop2 to function, then sqoop2 should take care of this step. Am I missing some step not mentioned on that page?
>
>
> { name : "Justin Lee",
> title : "Software Engineer",
> twitter : "@evanchooly",
> web : [ "10gen.com", "antwerkz.com" ],
> location : "New York, NY" }