You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Yingyuan Cheng <yi...@staff.sina.com.cn> on 2008/04/02 10:36:55 UTC

Help: libhdfs SIGSEGV

Hello.

Is libhdfs thread-safe? I can run single thread reading/writing HDFS
through libhdfs well, but when incrementing number of threads to 2 or
above, I received sigsegv error:

#
# An unexpected error has been detected by Java Runtime Environment:
#
# Internal Error (53484152454432554E54494D450E4350500214), pid=15614,
tid=1080834960
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode)
# An error report file with more information is saved as hs_err_pid15614.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
/bin/sh: line 1: 15614 Aborted ./hdfsbench -a w /tmp/test/txt -t 2


and the bt output:


(gdb) bt full
#0 0x061707b8 in ChunkPool::allocate () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#1 0x061703a6 in Arena::Arena () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#2 0x06501729 in Thread::Thread () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#3 0x06503144 in JavaThread::JavaThread () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0x062f006e in attach_current_thread () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0x062eee08 in jni_AttachCurrentThread () from
/usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xb7f1a4bd in getJNIEnv () at hdfsJniHelper.c:347
vm = (JavaVM *) 0x65ccfcc
vmBuf = 0xb74a0270
env = (JNIEnv *) 0xb7eb73e2
rv = 106745804
noVMs = 1
#7 0xb7f17143 in hdfsConnect (host=0x804aac2 "default", port=0) at
hdfs.c:119
env = (JNIEnv *) 0x0
jConfiguration = (jobject) 0x0
jFS = (jobject) 0x0
jURI = (jobject) 0x0
jURIString = (jstring) 0x0
jVal = {z = 0 '\0', b = 0 '\0', c = 0, s = 0, i = 0, j = 0, f = 0, d =
0, l = 0x0}
cURI = 0x0
gFsRef = (jobject) 0x0
#8 0x0804904b in worker_thread_w (arg=0x1) at hdfsbench.cpp:192
idx = 1
path = {static npos = 4294967295,
_M_dataplus = {<std::allocator<char>> =
{<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
_M_p = 0x804e484 "/tmp/test/txt1"}}
fs = (hdfsFS) 0x0
writeFile = (hdfsFile) 0x0
i = 0
bwrite = 0
boffset = 0
tv_start = {tv_sec = 0, tv_usec = 0}
tv_end = {tv_sec = 0, tv_usec = 0}
#9 0xb7f2246b in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#10 0xb7da06de in clone () from /lib/tls/i686/cmov/libc.so.6


--
Yingyuan Cheng

Re: Help: libhdfs SIGSEGV

Posted by Christian Kunz <ck...@yahoo-inc.com>.
Yingyuan ,

I cannot give you a detailed answer, just a guess.
Maybe Arun can chime in and provide more details.

My guess is that it has to do with the fact that the DFSClient returns the
same (cached) FileSystem handle for every connection request to the same
namenode, but libhdfs would return you a fresh global reference for every
call to hdfsConnect.
Your original code would work if libhdfs had a similar cache of global
references per Filesystem.

-Christian


On 4/2/08 11:31 PM, "Yingyuan Cheng" <yi...@staff.sina.com.cn> wrote:

> Hello Christian.
> 
> As you said, it does work. Thank you very much.
> 
> Would you like to explain further to me why it can work?
> 
> --
> 
> Yingyuan
> 
> 
> Christian Kunz 写道:
>> Hi Yingyuan,
>> 
>> Did you try to connect to hdfs in main function before spawning off the
>> threads and using the same handle in all the threads?
>> 
>> -Christian
>> 
>> 
>> Christian

Re: Help: libhdfs SIGSEGV

Posted by Yingyuan Cheng <yi...@staff.sina.com.cn>.
Hello Christian.

As you said, it does work. Thank you very much.

Would you like to explain further to me why it can work?

--

Yingyuan


Christian Kunz 写道:
> Hi Yingyuan,
>
> Did you try to connect to hdfs in main function before spawning off the
> threads and using the same handle in all the threads?
>
> -Christian
>
>
> Christian

Re: Help: libhdfs SIGSEGV

Posted by Christian Kunz <ck...@yahoo-inc.com>.
Hi Yingyuan,

Did you try to connect to hdfs in main function before spawning off the
threads and using the same handle in all the threads?

-Christian


On 4/2/08 6:21 PM, "Yingyuan Cheng" <yi...@staff.sina.com.cn> wrote:

> Hello Arun and all.
> 
> I generated a data block in my main function, then spawned serveral
> threads to write this block serveral times to different files in hdfs,
> my thread function as following:
> 
> -------------------------------------------------------------------
> // File: hdfsbench.cpp
> 
> void* worker_thread_w(void *arg)
> {
> int idx = (int)arg;
> std::string path = g_config.path_prefix;
> string_append(path, "%d", idx);
> 
> hdfsFS fs = hdfsConnect("default", 0);
> if(!fs) {
> PERRORL(ERROR, "Thread %d failed to connect to hdfs!\n", idx);
> g_workers->at(idx).status = TS_FAIL;
> return (void*)false;
> }
> 
> hdfsFile writeFile = hdfsOpenFile(fs, path.c_str(), O_WRONLY, 0, 0, 0);
> if(!writeFile) {
> PERRORL(ERROR, "Thread %d failed to open %s for writing!\n", idx,
> path.c_str());
> g_workers->at(idx).status = TS_FAIL;
> //hdfsDisconnect(fs);
> return (void*)false;
> }
> 
> int i;
> int bwrite, boffset;
> struct timeval tv_start, tv_end;
> 
> gettimeofday(&tv_start, NULL);
> 
> for (i = 0; i < g_config.num_blocks; i++) {
> boffset = 0;
> while (boffset < g_config.block_size) {
> bwrite = hdfsWrite(fs, writeFile, g_buffer->ptr + boffset,
> g_config.block_size - boffset);
> if (bwrite <0) {
> PERRORL(ERROR, "Thread %d failed when writing %s at block %d offset %d\n",
> idx, path.c_str(), i, boffset);
> g_workers->at(idx).status = TS_FAIL;
> i = g_config.num_blocks;
> break;
> }
> boffset += bwrite;
> g_workers->at(idx).nbytes += bwrite;
> }
> }
> 
> gettimeofday(&tv_end, NULL);
> hdfsCloseFile(fs, writeFile);
> //hdfsDisconnect(fs);
> g_workers->at(idx).elapsed = get_timeval_differ(tv_start, tv_end);
> 
> if (g_workers->at(idx).status == TS_UNSET ||
> g_workers->at(idx).status == TS_RUNNING) {
> g_workers->at(idx).status = TS_SUCCESS;
> }
> 
> return (void*)true;
> }
> 
> -------------------------------------------------------------------
> 
> And the following is my running script:
> 
> -------------------------------------------------------------------
> 
> #!/bin/bash
> #
> # File: hdfsbench.sh
> 
> HADOOP_HOME="/opt/hadoop"
> JAVA_HOME="/opt/java"
> 
> export 
> CLASSPATH="$HADOOP_HOME/hadoop-0.16.1-core.jar:$HADOOP_HOME/lib/commons-loggin
> g-1.0.4.jar:$HADOOP_HOME/lib/log4j-1.2.13.jar:$HADOOP_HOME/conf"
> export LD_LIBRARY_PATH="$HADOOP_HOME/libhdfs:$JAVA_HOME/lib/i386/server"
> 
> ./hdfsbench $@
> 
> -------------------------------------------------------------------
> 
> 
> Yingyuan
> 
> Arun C Murthy 写道:
>> 
>> On Apr 2, 2008, at 1:36 AM, Yingyuan Cheng wrote:
>> 
>>> Hello.
>>> 
>>> Is libhdfs thread-safe? I can run single thread reading/writing HDFS
>>> through libhdfs well, but when incrementing number of threads to 2 or
>>> above, I received sigsegv error:
>>> 
>> 
>> Could you explain a bit more? What are you doing in different threads?
>> Are you writing to the same file?
>> 
>> Arun
>> 
>

Re: Help: libhdfs SIGSEGV

Posted by Yingyuan Cheng <yi...@staff.sina.com.cn>.
Hello Arun and all.

I generated a data block in my main function, then spawned serveral 
threads to write this block serveral times to different files in hdfs, 
my thread function as following:

-------------------------------------------------------------------
// File: hdfsbench.cpp

void* worker_thread_w(void *arg)
{
int idx = (int)arg;
std::string path = g_config.path_prefix;
string_append(path, "%d", idx);

hdfsFS fs = hdfsConnect("default", 0);
if(!fs) {
PERRORL(ERROR, "Thread %d failed to connect to hdfs!\n", idx);
g_workers->at(idx).status = TS_FAIL;
return (void*)false;
}

hdfsFile writeFile = hdfsOpenFile(fs, path.c_str(), O_WRONLY, 0, 0, 0);
if(!writeFile) {
PERRORL(ERROR, "Thread %d failed to open %s for writing!\n", idx, 
path.c_str());
g_workers->at(idx).status = TS_FAIL;
//hdfsDisconnect(fs);
return (void*)false;
}

int i;
int bwrite, boffset;
struct timeval tv_start, tv_end;

gettimeofday(&tv_start, NULL);

for (i = 0; i < g_config.num_blocks; i++) {
boffset = 0;
while (boffset < g_config.block_size) {
bwrite = hdfsWrite(fs, writeFile, g_buffer->ptr + boffset,
g_config.block_size - boffset);
if (bwrite <0) {
PERRORL(ERROR, "Thread %d failed when writing %s at block %d offset %d\n",
idx, path.c_str(), i, boffset);
g_workers->at(idx).status = TS_FAIL;
i = g_config.num_blocks;
break;
}
boffset += bwrite;
g_workers->at(idx).nbytes += bwrite;
}
}

gettimeofday(&tv_end, NULL);
hdfsCloseFile(fs, writeFile);
//hdfsDisconnect(fs);
g_workers->at(idx).elapsed = get_timeval_differ(tv_start, tv_end);

if (g_workers->at(idx).status == TS_UNSET ||
g_workers->at(idx).status == TS_RUNNING) {
g_workers->at(idx).status = TS_SUCCESS;
}

return (void*)true;
}

-------------------------------------------------------------------

And the following is my running script:

-------------------------------------------------------------------

#!/bin/bash
#
# File: hdfsbench.sh

HADOOP_HOME="/opt/hadoop"
JAVA_HOME="/opt/java"

export 
CLASSPATH="$HADOOP_HOME/hadoop-0.16.1-core.jar:$HADOOP_HOME/lib/commons-logging-1.0.4.jar:$HADOOP_HOME/lib/log4j-1.2.13.jar:$HADOOP_HOME/conf"
export LD_LIBRARY_PATH="$HADOOP_HOME/libhdfs:$JAVA_HOME/lib/i386/server"

./hdfsbench $@

-------------------------------------------------------------------


Yingyuan

Arun C Murthy 写道:
>
> On Apr 2, 2008, at 1:36 AM, Yingyuan Cheng wrote:
>
>> Hello.
>>
>> Is libhdfs thread-safe? I can run single thread reading/writing HDFS
>> through libhdfs well, but when incrementing number of threads to 2 or
>> above, I received sigsegv error:
>>
>
> Could you explain a bit more? What are you doing in different threads? 
> Are you writing to the same file?
>
> Arun
>

Re: Help: libhdfs SIGSEGV

Posted by Arun C Murthy <ar...@yahoo-inc.com>.
On Apr 2, 2008, at 1:36 AM, Yingyuan Cheng wrote:

> Hello.
>
> Is libhdfs thread-safe? I can run single thread reading/writing HDFS
> through libhdfs well, but when incrementing number of threads to 2 or
> above, I received sigsegv error:
>

Could you explain a bit more? What are you doing in different  
threads? Are you writing to the same file?

Arun

> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> # Internal Error (53484152454432554E54494D450E4350500214), pid=15614,
> tid=1080834960
> #
> # Java VM: Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode)
> # An error report file with more information is saved as  
> hs_err_pid15614.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
> /bin/sh: line 1: 15614 Aborted ./hdfsbench -a w /tmp/test/txt -t 2
>
>
> and the bt output:
>
>
> (gdb) bt full
> #0 0x061707b8 in ChunkPool::allocate () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #1 0x061703a6 in Arena::Arena () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #2 0x06501729 in Thread::Thread () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #3 0x06503144 in JavaThread::JavaThread () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #4 0x062f006e in attach_current_thread () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #5 0x062eee08 in jni_AttachCurrentThread () from
> /usr/lib/jvm/java/jre/lib/i386/server/libjvm.so
> No symbol table info available.
> #6 0xb7f1a4bd in getJNIEnv () at hdfsJniHelper.c:347
> vm = (JavaVM *) 0x65ccfcc
> vmBuf = 0xb74a0270
> env = (JNIEnv *) 0xb7eb73e2
> rv = 106745804
> noVMs = 1
> #7 0xb7f17143 in hdfsConnect (host=0x804aac2 "default", port=0) at
> hdfs.c:119
> env = (JNIEnv *) 0x0
> jConfiguration = (jobject) 0x0
> jFS = (jobject) 0x0
> jURI = (jobject) 0x0
> jURIString = (jstring) 0x0
> jVal = {z = 0 '\0', b = 0 '\0', c = 0, s = 0, i = 0, j = 0, f = 0, d =
> 0, l = 0x0}
> cURI = 0x0
> gFsRef = (jobject) 0x0
> #8 0x0804904b in worker_thread_w (arg=0x1) at hdfsbench.cpp:192
> idx = 1
> path = {static npos = 4294967295,
> _M_dataplus = {<std::allocator<char>> =
> {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data  
> fields>},
> _M_p = 0x804e484 "/tmp/test/txt1"}}
> fs = (hdfsFS) 0x0
> writeFile = (hdfsFile) 0x0
> i = 0
> bwrite = 0
> boffset = 0
> tv_start = {tv_sec = 0, tv_usec = 0}
> tv_end = {tv_sec = 0, tv_usec = 0}
> #9 0xb7f2246b in start_thread () from /lib/tls/i686/cmov/ 
> libpthread.so.0
> No symbol table info available.
> #10 0xb7da06de in clone () from /lib/tls/i686/cmov/libc.so.6
>
>
> --
> Yingyuan Cheng
>