cygwin necessary for HDFS access from Windows?

[Robert Krüger <kr...@signal7.de>]

Hi,

I'm trying to run a program (basically a simple thruput benchmark, I've
already tested on Linux) in Eclipse on Windows. The HDFS is located on a
Linux server, though.

The exception I get is the following (probably the same Rutuja Joshi
posted earlier this month):

INFO: initializing HDFS with config Configuration: hadoop-default.xml,
hadoop-site.xml,
file:/C:/w/hadoop-test/./conf/roy-client/hadoop-default.xml,
file:/C:/w/hadoop-test/./conf/roy-client/hadoop-site.xml
08/07/15 18:36:08 WARN fs.FileSystem: uri=hdfs://roy:9000
javax.security.auth.login.LoginException: Login failed: CreateProcess:
whoami error=2
	at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
	at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
	at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
	at
org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1353)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1289)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
	at com.signal7.hadooptest.HadoopTest.getStorageSystem(HadoopTest.java:93)
	at com.signal7.hadooptest.HadoopTest.main(HadoopTest.java:31)
Exception in thread "main" java.io.IOException
	at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:175)
	at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
	at com.signal7.hadooptest.HadoopTest.getStorageSystem(HadoopTest.java:93)
	at com.signal7.hadooptest.HadoopTest.main(HadoopTest.java:31)
Caused by: javax.security.auth.login.LoginException: Login failed:
CreateProcess: whoami error=2
	at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
	at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
	at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:173)
	... 8 more


The hadoop-site.xml I'm using on the client side has dfs.permissions set
to false as well as the hadoop installation running the daemons on Linux.

I must be missing something. It can't be that I need cygwin to access
HDFS from a Java Program from Windows, right? The documentation says
"Cygwin - Required for shell support" but I'm not running a shell. If
so, are there any plans to get rid of that dependency?

Thanks in advance for any insight into this.

Regards,

Robert

Re: cygwin necessary for HDFS access from Windows?

[Steve Loughran <st...@apache.org>]

Robert Krüger wrote:
> Hi,
> 
> I'm trying to run a program (basically a simple thruput benchmark, I've
> already tested on Linux) in Eclipse on Windows. The HDFS is located on a
> Linux server, though.
> 
> The exception I get is the following (probably the same Rutuja Joshi
> posted earlier this month):
> 
> INFO: initializing HDFS with config Configuration: hadoop-default.xml,
> hadoop-site.xml,
> file:/C:/w/hadoop-test/./conf/roy-client/hadoop-default.xml,
> file:/C:/w/hadoop-test/./conf/roy-client/hadoop-site.xml
> 08/07/15 18:36:08 WARN fs.FileSystem: uri=hdfs://roy:9000
> javax.security.auth.login.LoginException: Login failed: CreateProcess:
> whoami error=2
> 	at


yes it is.

I think you could strip out cygwin support; right now it is needed at 
build and run time. But, someone would have to sit down and do the 
engineering needed to do this. Which, in the open source world, means 
someone who really wants to fix the problems has to put in the hours.

-steve