You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by yxxtdc <ya...@teradata.com> on 2011/03/08 19:50:25 UTC
Anybody can help with MountableHDFS fuse-dfs please?
Hi,
I am a new user to HDFS and am trying to following the instruction on:
http://wiki.apache.org/hadoop/MountableHDFS
But fuse-dfs fails to load and spits out the following errors:
fuse_dfs_wrapper.sh dfs://xsn95:19001 /hdfs -d
port=19001,server=xsn95
fuse-dfs didn't recognize /hdfs,-2
fuse-dfs ignoring option -d
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00080000
INIT: 7.8
flags=0x00000001
max_readahead=0x00080000
max_write=0x00020000
unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 40
Call to JNI_CreateJavaVM option 1 = -Xmx2048m
Calling JNI_CreateJavaVM
There was an error trying to initialize the HPI library.
Call to JNI_CreateJavaVM failed with error: -1
unique: 2, error: -5 (Input/output error), outsize: 16
I've added libnsl.so and libhpi.so to the LD_LIBRARY_PATH and still it won't
load. The error code is from:
Hadoop-0.20.2/src/c++/libhdfs/hdfsJniHelper.c
In getJNIEnv function:
//Create the VM
JavaVMInitArgs vm_args;
JavaVM *vm;
vm_args.version = JNI_VERSION_1_4;
vm_args.options = options;
vm_args.nOptions = noArgs;
vm_args.ignoreUnrecognized = 1;
fprintf(stderr, "Calling JNI_CreateJavaVM\n");
rv = JNI_CreateJavaVM(&vm, (void*)&env, &vm_args);
if (rv != 0) {
fprintf(stderr, "Call to JNI_CreateJavaVM failed "
"with error: %d\n", rv);
UNLOCK_JVM_MUTEX();
return NULL;
}
Anybody out there in HADOOP community can help me out please?
Thanks in advance.
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652043.html
Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Mar 8, 2011, at 3:41 PM, yxxtdc wrote:
> Thanks Brian.
>
> Got over that by fixing a typo in LD_LIBRARY_PATH and now fuse_dfs is
> mounting albeit very very very slowly, like one inode a minute.
>
What do you mean by "mounting"? Do you mean "listing"?
Anything on the order of a minute is out of the ordinary. See if you can remove some of the "moving pieces" and try to replicate with the stock "hadoop" client.
FUSE, especially on Hadoop 0.20, is not a rocketship when it comes to metadata operations. However, it's not horrid. See below for example.
Brian
[bbockelm@t3-sl5 ~]$ time ls -1 /mnt/hadoop/dropfiles | wc -l
22
real 0m0.135s
user 0m0.001s
sys 0m0.003s
[bbockelm@t3-sl5 ~]$ time ls -1 /tmp | wc -l
65
real 0m0.002s
user 0m0.000s
sys 0m0.002s
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by yxxtdc <ya...@teradata.com>.
Thanks Brian.
Got over that by fixing a typo in LD_LIBRARY_PATH and now fuse_dfs is
mounting albeit very very very slowly, like one inode a minute.
Yang
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652761.html
Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by Brian Bockelman <bb...@cse.unl.edu>.
I would avoid making a manual copy of libjvm.so - see if you can figure out why adding the JDK copy didn't work. For reference, here's what we use:
if [ "$LD_LIBRARY_PATH" = "" ]
then JVM_LIB=`find ${JAVA_HOME}/jre/lib -name libjvm.so |tail -n 1`
export LD_LIBRARY_PATH=`dirname $JVM_LIB`:/usr/lib/
fi
Brian
On Mar 8, 2011, at 1:38 PM, yxxtdc wrote:
> I copied the libjvm.so from
> /usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so.
> I added /usr/java/jdk1.6.0_24/jre/lib/amd64/server to the LD_LIBRARY_PATH
> and it did not work so I made a manual copy to $HADOOP_HOME/lib directory.
>
> I googled my HPI error someone suggested to use the client version of the
> libjvm.so but jdk 6 only came with the lib/amd64/server/libjvm.so.
>
>
> Thanks,
>
> Yang
>
>
> --
> View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652276.html
> Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by yxxtdc <ya...@teradata.com>.
I copied the libjvm.so from
/usr/java/jdk1.6.0_24/jre/lib/amd64/server/libjvm.so.
I added /usr/java/jdk1.6.0_24/jre/lib/amd64/server to the LD_LIBRARY_PATH
and it did not work so I made a manual copy to $HADOOP_HOME/lib directory.
I googled my HPI error someone suggested to use the client version of the
libjvm.so but jdk 6 only came with the lib/amd64/server/libjvm.so.
Thanks,
Yang
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652276.html
Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Mar 8, 2011, at 1:04 PM, yxxtdc wrote:
> Hi,
>
> xsn95:/ # ldd /root/build_src/hadoop-0.20.2/lib/libhdfs.so.0
> linux-vdso.so.1 => (0x00007ffff413c000)
> libjvm.so => /root/build_src/hadoop-0.20.2/lib/libjvm.so
> (0x00002adbd5442000)
This is likely problematic. Why is libjvm.so here, instead of the normal place inside the JVM install? I'm wondering if maybe the JVM is looking for libhpi.so relative to your libjvm.
> libdl.so.2 => /lib64/libdl.so.2 (0x00002adbd5f62000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002adbd6066000)
> libc.so.6 => /lib64/libc.so.6 (0x00002adbd617f000)
> libm.so.6 => /lib64/libm.so.6 (0x00002adbd63c0000)
> /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
>
>
> It does not look like hdfs_test is built along with libhdfs.so. Is there a
> compile flag I need to turn on with ant to compile hdfs_test executable?
> Sorry, I am totally new at this and still learning. Any help would be deeply
> appreciated.
>
> Thanks,
>
> Yang
>
>
> --
> View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652121.html
> Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by yxxtdc <ya...@teradata.com>.
Hi,
xsn95:/ # ldd /root/build_src/hadoop-0.20.2/lib/libhdfs.so.0
linux-vdso.so.1 => (0x00007ffff413c000)
libjvm.so => /root/build_src/hadoop-0.20.2/lib/libjvm.so
(0x00002adbd5442000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002adbd5f62000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002adbd6066000)
libc.so.6 => /lib64/libc.so.6 (0x00002adbd617f000)
libm.so.6 => /lib64/libm.so.6 (0x00002adbd63c0000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
It does not look like hdfs_test is built along with libhdfs.so. Is there a
compile flag I need to turn on with ant to compile hdfs_test executable?
Sorry, I am totally new at this and still learning. Any help would be deeply
appreciated.
Thanks,
Yang
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652121.html
Sent from the Users mailing list archive at Nabble.com.
Re: Anybody can help with MountableHDFS fuse-dfs please?
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hi,
Sounds like an issue with your Hadoop runtime environment.
What does "ldd </path/to/libhdfs.so>" say? What happens if you try one of the libhdfs test applications?
Brian
On Mar 8, 2011, at 12:50 PM, yxxtdc wrote:
> Hi,
>
> I am a new user to HDFS and am trying to following the instruction on:
> http://wiki.apache.org/hadoop/MountableHDFS
>
> But fuse-dfs fails to load and spits out the following errors:
> fuse_dfs_wrapper.sh dfs://xsn95:19001 /hdfs -d
> port=19001,server=xsn95
> fuse-dfs didn't recognize /hdfs,-2
> fuse-dfs ignoring option -d
> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000003
> max_readahead=0x00080000
> INIT: 7.8
> flags=0x00000001
> max_readahead=0x00080000
> max_write=0x00020000
> unique: 1, error: 0 (Success), outsize: 40
> unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 40
> Call to JNI_CreateJavaVM option 1 = -Xmx2048m
> Calling JNI_CreateJavaVM
> There was an error trying to initialize the HPI library.
> Call to JNI_CreateJavaVM failed with error: -1
> unique: 2, error: -5 (Input/output error), outsize: 16
>
>
>
> I've added libnsl.so and libhpi.so to the LD_LIBRARY_PATH and still it won't
> load. The error code is from:
> Hadoop-0.20.2/src/c++/libhdfs/hdfsJniHelper.c
>
> In getJNIEnv function:
>
> //Create the VM
>
> JavaVMInitArgs vm_args;
>
> JavaVM *vm;
>
> vm_args.version = JNI_VERSION_1_4;
>
> vm_args.options = options;
>
> vm_args.nOptions = noArgs;
>
> vm_args.ignoreUnrecognized = 1;
>
>
>
> fprintf(stderr, "Calling JNI_CreateJavaVM\n");
>
> rv = JNI_CreateJavaVM(&vm, (void*)&env, &vm_args);
>
> if (rv != 0) {
>
> fprintf(stderr, "Call to JNI_CreateJavaVM failed "
>
> "with error: %d\n", rv);
>
> UNLOCK_JVM_MUTEX();
>
> return NULL;
>
> }
>
>
>
> Anybody out there in HADOOP community can help me out please?
>
> Thanks in advance.
>
>
> --
> View this message in context: http://hadoop-common.472056.n3.nabble.com/Anybody-can-help-with-MountableHDFS-fuse-dfs-please-tp2652043p2652043.html
> Sent from the Users mailing list archive at Nabble.com.