Minicluster vs NFS gateway

[Adam Soroka <aj...@apache.org>]

Hello Hadoop folks! I am trying to use the NFS gateway component with a Minicluster under version 3.1.0, but I cannot start the gateway using `hdfs nfs3`. I'm running into an exception that puzzles me:

Exception in thread "main" java.nio.file.FileSystemException: Only HDFS is supported as underlyingFileSystem, fs scheme:file uri to be addedfile:///
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getResolvedURI(Nfs3Utils.java:268)
	at org.apache.hadoop.hdfs.nfs.nfs3.DFSClientCache.prepareAddressMap(DFSClientCache.java:167)
	at org.apache.hadoop.hdfs.nfs.nfs3.DFSClientCache.<init>(DFSClientCache.java:144)
	at org.apache.hadoop.hdfs.nfs.nfs3.DFSClientCache.<init>(DFSClientCache.java:138)
	at org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.<init>(RpcProgramNfs3.java:189)
	at org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.createRpcProgramNfs3(RpcProgramNfs3.java:227)
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.<init>(Nfs3.java:45)
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:67)
	at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.main(Nfs3.java:79)

From my reading of the code, this shouldn't be possible. The Minicluster exposes a MiniDFS filesystem, and when I look at the storage on-disk, it does indeed seem to be an HDFS layout. As far as I can tell, this line:

https://github.com/apache/hadoop/blob/rel/release-3.1.0/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java#L1333

means that the URI for the filesystem is hard-coded and the protocol must be hdfs:, not file:. 

Am I missing something important? Is it not possible to use the NFS gateway with the Minicluster? The tests for NFS seem to do just that:

https://github.com/apache/hadoop/blob/rel/release-3.1.0/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestMountd.java

but I am a Hadoop neophyte and maybe I misunderstand how to use it. I'm launching with the parameters `-nnport 8020 -rmport 8032 -jhsport 10020`. I have tried launching the Minicluster without the `-namenode` argument and with it set to `hdfs://localhost:8020`. In either case I get a perfectly nice HDFS service at port 8020, but the darn NFS gateway just doesn't seem to notice!

Thank you for any help or advice!

Adam Soroka

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org