You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Phillip Rhodes <pr...@mammothdata.com> on 2015/05/18 23:58:59 UTC
Sqoop on Windows "Not a valid DFS filename"
Sqoop crew:
I'm trying to use Sqoop 1.4.5 on Windows and feel like I'm
*almost* there. But I've hit one stumbling block I can't
yet figure out and am hoping someone can point me in
the right direction.
Right now, I can get Sqoop to run and it gets as far as generating
the Java class for the table I'm importing, and then
starts trying to push jar files to the jar file cache. And this
is where the trouble comes in. Since I'm on Windows
the path to those jars is something like
e:\win-hadoop\hadoop-3.0.0-SNAPSHOT\sqoop\lib\db2jcc4.jar
and the like. This of course results in errors like this:
15/05/18 17:36:21 ERROR tool.ImportTool: Imported Failed: Pathname
/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar from hdfs://
10.1.115.231:9000/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar
is not a valid DFS filename.
Any thoughts on how to get around that? This happens whether I'm running
the sqoop.cmd
file in a regular Windows cmd window, or running the unix script in a
Cygwin shell.
(Also, if anybody is wondering, this Hadoop 3 business is because I built
Hadoop from
source to get the Windows support and building HEAD just gives you stuff
labeled Hadoop 3. I don't *think* that's part of the problem, but feel
free to convince
me otherwise).
Thanks,
Phillip Rhodes
Re: Sqoop on Windows "Not a valid DFS filename"
Posted by Phillip Rhodes <pr...@mammothdata.com>.
On Tue, May 19, 2015 at 1:19 AM, Xu, Qian A <qi...@intel.com> wrote:
> Hi Phillip,
>
>
>
> Working on Windows via Cygwin is quite interesting idea.
>
> · Could you post the whole command you are using?
>
> · Is this jar expected to be added to classpath?
>
>
>
> Maybe you can try to change
> “E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar” to
> “/E/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar”.
>
>
>
I don't have control over that path, it's something Sqoop does internally.
I'm assuming it works out which jars need to be pushed to HDFS for use by
the map-reduce job and then just takes them from the Sqoop classpath and
tries to push them up to an identical path in HDFS.
Unfortunately, changing everything to use unix style paths under Cygwin (by
changing SQOOP_HOME, etc.) doesn't work either, because then Java doesn't
know how to find the jars as it's a native Windows program and ultimately
passes those strings to some Windows API call that doesn't understand
/cygdrive/e/foo/bar/whatever
So far, short of hacking up the Sqoop source, I can't find a way to make
this work, but I'm open to suggestions...
Phil
Re: Sqoop on Windows "Not a valid DFS filename"
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
If you want to use Sqoop on windows, you can use the windows distribution from Hortonworks. Please see http://hortonworks.com/hdp/downloads/# Microsoft also contributed a few patches towards windows support into Sqoop (and in Hadoop in general)
Thanks
Venkat
From: <Xu>, Qian A
Reply-To: "user@sqoop.apache.org<ma...@sqoop.apache.org>"
Date: Monday, May 18, 2015 at 10:19 PM
To: "user@sqoop.apache.org<ma...@sqoop.apache.org>"
Subject: RE: Sqoop on Windows "Not a valid DFS filename"
Hi Phillip,
Working on Windows via Cygwin is quite interesting idea.
· Could you post the whole command you are using?
· Is this jar expected to be added to classpath?
Maybe you can try to change “E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar” to “/E/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar”.
Thanks
Stanley (Xu, Qian)
From: Phillip Rhodes [mailto:prhodes@mammothdata.com]
Sent: Tuesday, May 19, 2015 5:59 AM
To: user@sqoop.apache.org<ma...@sqoop.apache.org>
Subject: Sqoop on Windows "Not a valid DFS filename"
Sqoop crew:
I'm trying to use Sqoop 1.4.5 on Windows and feel like I'm
*almost* there. But I've hit one stumbling block I can't
yet figure out and am hoping someone can point me in
the right direction.
Right now, I can get Sqoop to run and it gets as far as generating
the Java class for the table I'm importing, and then
starts trying to push jar files to the jar file cache. And this
is where the trouble comes in. Since I'm on Windows
the path to those jars is something like
e:\win-hadoop\hadoop-3.0.0-SNAPSHOT\sqoop\lib\db2jcc4.jar
and the like. This of course results in errors like this:
15/05/18 17:36:21 ERROR tool.ImportTool: Imported Failed: Pathname /E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar from hdfs://10.1.115.231:9000/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar<http://10.1.115.231:9000/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar> is not a valid DFS filename.
Any thoughts on how to get around that? This happens whether I'm running the sqoop.cmd
file in a regular Windows cmd window, or running the unix script in a Cygwin shell.
(Also, if anybody is wondering, this Hadoop 3 business is because I built Hadoop from
source to get the Windows support and building HEAD just gives you stuff
labeled Hadoop 3. I don't *think* that's part of the problem, but feel free to convince
me otherwise).
Thanks,
Phillip Rhodes
RE: Sqoop on Windows "Not a valid DFS filename"
Posted by "Xu, Qian A" <qi...@intel.com>.
Hi Phillip,
Working on Windows via Cygwin is quite interesting idea.
· Could you post the whole command you are using?
· Is this jar expected to be added to classpath?
Maybe you can try to change “E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar” to “/E/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar”.
Thanks
Stanley (Xu, Qian)
From: Phillip Rhodes [mailto:prhodes@mammothdata.com]
Sent: Tuesday, May 19, 2015 5:59 AM
To: user@sqoop.apache.org
Subject: Sqoop on Windows "Not a valid DFS filename"
Sqoop crew:
I'm trying to use Sqoop 1.4.5 on Windows and feel like I'm
*almost* there. But I've hit one stumbling block I can't
yet figure out and am hoping someone can point me in
the right direction.
Right now, I can get Sqoop to run and it gets as far as generating
the Java class for the table I'm importing, and then
starts trying to push jar files to the jar file cache. And this
is where the trouble comes in. Since I'm on Windows
the path to those jars is something like
e:\win-hadoop\hadoop-3.0.0-SNAPSHOT\sqoop\lib\db2jcc4.jar
and the like. This of course results in errors like this:
15/05/18 17:36:21 ERROR tool.ImportTool: Imported Failed: Pathname /E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar from hdfs://10.1.115.231:9000/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar<http://10.1.115.231:9000/E:/win-hadoop/hadoop-3.0.0-SNAPSHOT/sqoop/lib/db2jcc4.jar> is not a valid DFS filename.
Any thoughts on how to get around that? This happens whether I'm running the sqoop.cmd
file in a regular Windows cmd window, or running the unix script in a Cygwin shell.
(Also, if anybody is wondering, this Hadoop 3 business is because I built Hadoop from
source to get the Windows support and building HEAD just gives you stuff
labeled Hadoop 3. I don't *think* that's part of the problem, but feel free to convince
me otherwise).
Thanks,
Phillip Rhodes