About fuse-dfs and NFS

[XuChris <yo...@hotmail.com>]

Hi,
 
I mount hdfs to a directory of localhost by fuse-dfs, and then export the directory. 
When access  the directory by NFS, I can read data of the directory,
but cannot write data to the directory. Why? 
Now I want to know does fuse-dfs support data-writing operation by NFS or not?
Who can help me? Thank you very much. 
 
My system configure:
OS: Fedora release 8(kernel 2.6.23.1)
      For NFS, the version of fuse module has updated to 2.7.4
Fuse:2.7.4
Hadoop:0.19.1
 
Best regards.
 
Chris 
2009-6-26
_________________________________________________________________
打工，挣钱，买房子，快来MClub一起”金屋藏娇”！
http://club.msn.cn/?from=10

Re: fuse-dfs then samba mount

[Todd Lipcon <to...@cloudera.com>]

On Thu, Aug 13, 2009 at 12:04 AM, Manhee Jo <jo...@nttdocomo.com> wrote:

> Hi all,
>
> I've succeeded in sharing hdfs files from windows xp through fuse-dfs then
> samba mount.
> When I tried to copy (read and write) 1GB text file from fuse-dfs over
> samba, it took around 50 secs.
> Then, I tried "dfs get" the same file to a data node's local file system
> and tried to copy the file
> from the data node (without fuse-dfs this time) over samba, again, which
> took around 30 seconds.
> Since the disk reads are paralleled and distributed, should it be faster
> then reading from one node?


Nope - the file is stored distributed, but a single reader (using dfs -get
or the DFSClient API from Java) won't do a parallel read from multiple
replicas. What you've seen seems about right - there's a measurable overhead
of going through the datanode compared to just using local disk.


>
> Well, I know it must depend on the file size. So then, here is my question.
> What is actually happening in fuse-dfs read? and samba?


It's a single connection to one datanode at a time. At the end of each
block, it connects to the DN that stores the next block and reads from that
one. At no time does it transfer in parallel from multiple replicas. Some
people have mentioned this as a feature request but it hasn't been
prioritized high yet for a multitude of reasons.

-Todd

fuse-dfs then samba mount

[Manhee Jo <jo...@nttdocomo.com>]

Hi all,

I've succeeded in sharing hdfs files from windows xp through fuse-dfs then 
samba mount.
When I tried to copy (read and write) 1GB text file from fuse-dfs over 
samba, it took around 50 secs.
Then, I tried "dfs get" the same file to a data node's local file system and 
tried to copy the file
from the data node (without fuse-dfs this time) over samba, again, which 
took around 30 seconds.
Since the disk reads are paralleled and distributed, should it be faster 
then reading from one node?
Well, I know it must depend on the file size. So then, here is my question.
What is actually happening in fuse-dfs read? and samba?
What is the threshold of the file size when fuse-dfs might win?


Thanks,
Manhee

Re: About fuse-dfs and NFS

[Brian Bockelman <bb...@cse.unl.edu>]

Hey Chris,

FUSE in general does not support NFS mounts well because it has a  
tendency to renumber inodes upon NFS restart, which causes clients to  
choke.

FUSE-DFS supports a limited range of write operations; it's possible  
that your application is trying to use write functionality that is not  
supported.

Brian

On Jun 26, 2009, at 2:57 AM, XuChris wrote:

>
>
> Hi,
>
> I mount hdfs to a directory of localhost by fuse-dfs, and then  
> export the directory.
> When access  the directory by NFS, I can read data of the directory,
> but cannot write data to the directory. Why?
> Now I want to know does fuse-dfs support data-writing operation by  
> NFS or not?
> Who can help me? Thank you very much.
>
> My system configure:
> OS: Fedora release 8(kernel 2.6.23.1)
>      For NFS, the version of fuse module has updated to 2.7.4
> Fuse:2.7.4
> Hadoop:0.19.1
>
> Best regards.
>
> Chris
> 2009-6-26
> _________________________________________________________________
> 打工，挣钱，买房子，快来MClub一起”金屋藏娇”！
> http://club.msn.cn/?from=10