RE: NFS Gateway - Secure Cluster - Mount Failed

["kumar, Senthil(AWF)" <se...@ebay.com>]

+ Hadoop Users.

--Senthil
From: Senthil Kumar [mailto:senthilec566@gmail.com]
Sent: Monday, August 29, 2016 4:22 PM
To: hdfs-dev@hadoop.apache.org
Cc: Senthil kumar <se...@gmail.com>; Maliakkal Padmanabhan, Aroop <am...@ebay.com>; kumar, Senthil(AWF) <se...@ebay.com>
Subject: Re: NFS Gateway - Secure Cluster - Mount Failed

Anybody facing the  issue in Secure Cluster ?? ..

added root directory in /etc/exports
 cat /etc/exports
/ *(rw,fsid=0,no_root_squash)

mount -vvv -t nfs -o nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,sync host:/ /hdfs_space
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "phxdpehdc30dn0007.stratus.phx.ebay.com:/"
mount: node:  "/hdfs_space"
mount: types: "nfs"
mount: opts:  "nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,sync"
final mount options: 'nfsvers=3,sec=krb5,proto=tcp,nolock,noacl'
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "host:/"
mount: external mount: argv[2] = "/hdfs_space"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sync,nfsvers=3,sec=krb5,proto=tcp,nolock,noacl"
mount.nfs: timeout set for Mon Aug 29 03:51:30 2016
mount.nfs: trying text-based options 'nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,addr=10.115.22.46'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.115.22.46 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.115.22.46 prog 100005 vers 3 prot TCP port 4242
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting host:/


--Senthil

On Thu, Aug 25, 2016 at 4:58 PM, Senthil Kumar <se...@gmail.com>> wrote:
Started NFS Service in DEBUG mode and found below logs ...

2016-08-25 03:59:05,766 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS NULL
2016-08-25 03:59:05,768 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT NULLOP :  client: /IP_ADDR
2016-08-25 03:59:05,770 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT NULLOP :  client: /IP_ADDR
2016-08-25 03:59:05,771 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT MNT path: / client: /IP_ADDR
2016-08-25 03:59:05,771 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: Got host: gateway path: /
2016-08-25 03:59:05,783 INFO org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: Giving handle (fileId:16385) to client for export /

==== { Looks like mount operation done } ======
2016-08-25 03:59:05,784 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT UMNT path: / client: /IP_ADDR
==== { Why client is Sending UMNT request } ====

Here is the MNT CMD:
 mount -vvv -t nfs -o vers=3,sec=krb5,proto=tcp,nolock,sync IP_ADDR:/ /hdfs_space

Can someone help me here to understand the behavior ?? and how to solve this mnt issue ??

--Senthil

On Thu, Aug 25, 2016 at 12:07 PM, Senthil Kumar <se...@gmail.com>> wrote:
Expected Client Kerberos Principle is null issue resolved now .. Added sec=krb5 option while mounting ..

mount -vvv -t nfs -o vers=3,sec=krb5,proto=tcp,nolock,noacl,sync gateway:/ hdfs_space/


mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "gatewaymachine:/"
mount: node:  "hdfs_space/"
mount: types: "nfs"
mount: opts:  "vers=3,sec=krb5,proto=tcp,nolock,noacl,sync"
final mount options: 'vers=3,sec=krb5,proto=tcp,nolock,noacl'
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "gatewaymachine:/"
mount: external mount: argv[2] = "hdfs_space/"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sync,vers=3,sec=krb5,proto=tcp,nolock,noacl"
mount.nfs: timeout set for Wed Aug 24 23:34:31 2016
mount.nfs: trying text-based options 'vers=3,sec=krb5,proto=tcp,nolock,noacl,addr=10.115.22.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.115.22.109 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.115.22.109 prog 100005 vers 3 prot TCP port 4242
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting gatewaymachine:/


Not sure why mount throwing permission issue .. Anybody faced this issue ??


--Senthil

On Thu, Aug 25, 2016 at 10:53 AM, Senthil Kumar <se...@gmail.com>> wrote:
Hi Team ,  As part of NFS Evaluation , i have installed NFS Gateway Service in Secure Cluster ..


Config in Gateway Machine:
<property>
        <name>nfs.file.dump.dir</name>
        <value>/tmp/.hdfs-nfs</value>
   </property>
   <property>
        <name>nfs.keytab.file</name>
        <value>/etc/hadoop/hadoop.keytab</value>
   </property>
   <property>
        <name>nfs.kerberos.principal</name>
        <va...@APD.XXXX.COM></value>
   </property>


NFS3 Service Started Successfully , but when i try to Mount the root / directory it failed with below error ..

WARN org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: Exception
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User root (auth:PROXY) via hadoop/phxdpehdc30dn0029.stratus.phx.ebay.com@APD.EBAY.COM<ma...@APD.EBAY.COM> (auth:KERBEROS)
 is not authorized for protocol interface org.apache.hadoop.hdfs.protocol.ClientProtocol, expected client Kerberos principal is null


mount command:
mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync gatewaymachine:/ hdfs_space/
mount.nfs: mount system call failed


What could be the issue here ??  I followed https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html this documentation ..


--Senthil