You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by Aastha Mehta <aa...@gmail.com> on 2011/03/01 18:39:19 UTC

problems with fuse-dfs

Hello,

I am facing problems in running fuse-dfs over hdfs. I came across this
thread while searching for my problem:
http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg00341.html
OR
http://search-hadoop.com/m/T1Bjv17q0eF1&subj=Re+Fuse+DFS

and it exactly mentions some of the symptoms I am looking at.

To quote Eli,
"fuse_impls_getattr.c connects via hdfsConnectAsUser so you should see a log
(unless its returning from a case that doesn't print an error). Next
step is to determine that you're actually reaching the code you modified by
adding a syslog to the top of the function (need to make sure you're
actually loading the libhdfs you've built vs an older one or another one
installed on your system), and then determine which error case in that
function you're seeing. It's strange that -d would cause that path to
change."

I get the error from fuse_impls_getattr.c that could not connect to my host
and port. I checked the syslogs and only this error is printed there. No
error is printed from hdfsConnectAsUser(). I even wrote down syslog
functions in hdfsConnectAsUser, but it is not printed. I used the following
commands to compile libhdfs and then fuse-dfs from $HADOOP_HOME:

$ export
LD_LIBRARY_PATH=$HADOOP_HOME/build/libhdfs:/usr/lib/jvm/java-1.5.0-sun:
/usr/lib/jvm/java-1.5.0-sun-1.5.0.19/include:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/i386/client
$ ant compile-c++-libhdfs -Dlibhdfs=1
$ ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
-Djava5.home=/usr/lib/jvm/java-1.5.0-sun-1.5.0.19
$ mkdir /media/myDrive/newhdfs
$ cd $HADOOP_HOME/build/contrib/fuse-dfs/
$ sudo sh fuse_dfs_wrapper.sh dfs://aastha-desktop:9000
/media/myDrive/newhdfs

And when i do:
$ sudo cat /var/log/syslog

at the end, I get
aastha-desktop fuse_dfs: ERROR: could not connect to aastha-desktop:9000,
fuse_impls_getattr.c:37

EIO value is 5, which I understand implies an Input/output error.

When i try to ls /media/myDrive/newhdfs, i get the input/output error.
Also if i try to unmount newhdfs, it says that it is in use and cannot be
unmounted.

I even tried -d option, but that just runs into an infinite loop of
connection retries and shows some socketconnection exception.

I have been working with fuse-dfs and hdfs for more than a month now. Till
two days back I could atleast create fuse-dfs instances and mount them using
the wrapper script. I could also create files inside it, modify them, etc.
and view the changes being updated in the nodes of the hadoop cluster. Since
yesterday, I am trying to create a LUN image inside fuse-dfs (using dd
if=/dev/zero of=... command), and check whether it is created properly on
the cluster or not. Yesterday, a normal file system was working and when I
tried to create a LUN, my syslog showed that an error was coming from
fuse_impls_open.c. But now a normal file system instance is also not getting
created and mounted properly.

Following are my questions based on this observation:
1. Please tell me, why is the fuse_impls_getattr.c error is coming now and
how can it be resolved.
2. Does ant compile-c++-libhdfs compile libhdfs? Will the changes that I
make in source files in libhdfs be updated by the command? Since I can't see
any syslog statements getting printed from hdfsConnectAsUser(), I am not
sure if libhdfs has been recompiled. Also I cannot see any object files
created inside libhdfs, so greater is my doubt.
3. How does libhdfs.so and libhdfs.so.0 come into picture in all this? We
have provided links to them in build/libhdfs; how do they work?

Any help with these issues would be highly appreciated.

Thanks,
Aastha.

--
Aastha Mehta
Intern, NetApp, Bangalore
4th year undergraduate, BITS Pilani
E-mail: aasthakm@gmail.com