You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Keith Wiley <kw...@keithwiley.com> on 2011/01/28 02:46:14 UTC
Thread safety issues with JNI/native code from map tasks?
I am seeing very perplexing segfaults and standard allocation exceptions in my native code (.so files passed to the distributed cace) which is called via JNI from the map task. This code runs perfectly fine (on the same data) outside Hadoop. Even when run in a Hadoop standalone mode (no cluster), it still segfaults. The memory footprint is quite small and inspection at run time reveals there is plenty of memory left, yet I get segfaults and exceptions.
I'm starting to wonder if this is a thread issue.
The native code is not *specifically* thread safe (not compiled with pthreads or anything like that).
However, it is also not run in any concurrent fashion except w.r.t. to the JVM itself. For example, my map task doesn't make parallel calls through JNI to the native code on concurrent threads at the Java level, nor does the native code itself spawn any threads (like I said, it isn't even compiled with pthreads).
However, there are clearly other "threads" of execution. For example, the JVM itself is running, including whatever supplemental threads the JVM involves (the garbage collector?). In addition, my Java mapper is running two Java threads at the time of the native code. One calls the native code and effectively blocks until the native code returns through JNI. The other just spins and sends reports and statuses to the job tracker at regular intervals to prevent the task from being killed, but it doesn't do anything else particularly memory-related, certainly no JNI/native calls, it's very basic, just sleep 'n report, sleep 'n report.
So, the question is, in the scenario I have described, is there any reason to suspect that the cause of my problems is some sort of thread trampling between the native code and something else in the surrounding environment (the JVM or something like that), especially in the context of the surrounding Hadoop infrastructure? It doesn't really make any sense to me, but I'm running out of ideas.
I've experimented with "mapred.child.java.opts" and "mapred.child.ulimit" but nothing really seems to have any effect on the frequency of these errors.
I'm quite out of ideas. These segfaults and standard allocation exceptions (in the face of plenty of free memory) have basically brought my work to a halt and I just don't know what to do anymore.
Thanks.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"And what if we picked the wrong religion? Every week, we're just making God
madder and madder!"
-- Homer Simpson
________________________________________________________________________________
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Keith Wiley <kw...@keithwiley.com>.
Hmmm, none of my native code is using JNI objects, so the memory in question should have no relationship to the Java heap or any other aspect of the Java world, but I admit, I'm unclear on who Java, JNI, and native libraries divide memory between one another or how they trade responsibility for allocating and deallocating memory.
I'll consider it, but since I'm not talking about JNI objects, I don't that can be it. Do you think I'm misunderstanding something?
On Jan 28, 2011, at 16:24 , Todd Lipcon wrote:
> JNI may also work fine with no GC running, but then work badly when GC kicks
> in at a bad time. For example, if you grab a pointer to a String or array,
> you need to essentially lock them so the GC doesn't relocate the objects
> underneath you. For example, maybe you're releasing one of these references
> and then continuing to use it?
>
> -Todd
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"Luminous beings are we, not this crude matter."
-- Yoda
________________________________________________________________________________
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Greg Roelofs <ro...@yahoo-inc.com>.
Todd Lipcon wrote:
> JNI may also work fine with no GC running, but then work badly when GC kicks
> in at a bad time. For example, if you grab a pointer to a String or array,
> you need to essentially lock them so the GC doesn't relocate the objects
> underneath you. For example, maybe you're releasing one of these references
> and then continuing to use it?
Excellent point! And one I should have remembered, too.
Keith, take a look at the native ZlibCompressor interface to see one way of
handling this. (It pins a number of buffers in memory and puts them into a
pool, if I remember Chris Douglas's explanation correctly. I didn't need to
dive into that level of detail myself for what I was working on, so I never
touched the buffer code and might not be remembering it entirely accurately,
but that's the gist, anyway.)
On 0.20.x or 0.22/trunk:
hadoop-common/src/native/src/org/apache/hadoop/io/compress
hadoop-common/src/{core or java}/org/apache/hadoop/io/compress
Greg
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Todd Lipcon <to...@cloudera.com>.
JNI may also work fine with no GC running, but then work badly when GC kicks
in at a bad time. For example, if you grab a pointer to a String or array,
you need to essentially lock them so the GC doesn't relocate the objects
underneath you. For example, maybe you're releasing one of these references
and then continuing to use it?
-Todd
On Fri, Jan 28, 2011 at 3:50 PM, Greg Roelofs <ro...@yahoo-inc.com> wrote:
> Keith Wiley wrote:
>
> > (1) Speculative execution would occur on a completely different
> > node, so there definitely isn't any thread cross-talk (in memory).
> > So long as they don't rely on reading/writing temp files from
> > HDFS I don't see how they could have any effect on one another.
>
> Good point.
>
> > (2) I am also getting seg faults when I run in noncluster
> > standalone mode, which is a single nonspeculated thread......I
> > presume.
>
> That's the same as "pseudo-distributed mode"?
>
> > Can you explain your thoughts on speculative execution w.r.t. the
> > problems I'm having?
>
> Thoughts? You expect me to have thoughts, too??
>
> :-)
>
> I had not fully thought through the spec ex idea; it was the only thing
> I could think of that might put two (otherwise independent) JNI-using tasks
> onto the same node. But as you point out above, it wouldn't...
>
> Does your .so depend on any other potentially thread-unsafe .so that other
> (non-Hadoop) processes might be using? System libraries like zlib are safe
> (else they wouldn't make very good system libraries), but maybe some other
> research library or something? (That's a long shot, but I'm pretty much
> grasping at straws here.)
>
> > Yes, not thread safe, but what difference could that make if I
> > don't use the library in a multi-threaded fashion. One map task,
> > one node, one Java thread calling JNI and using the native code?
> > How do thread safety issues factor into this? I admit, it's
> > my theory that threads might be involved somehow, but I don't
> > understand how, I'm just shooting in the dark since I can't
> > solve this problem any other way yet.
>
> Since you can reproduce it in standalone mode, can you enable core dumps
> so you can see the backtrace of the code that segfaults? Knowing what
> specifically broke and how it got there is always a big help.
>
> Btw, keep in mind that there are memory-related bugs that don't show up
> until there's something big in memory that pushes the code in question
> up into a region with different data patterns in it (most frequently zero
> vs. non-zero, but others are possible). IOW, maybe the code is dependent
> on uninitialized memory, but you were getting lucky when you ran it outside
> of Hadoop. Have you run it through valgrind or Purify or similar?
>
> Greg
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Jan 28, 2011, at 5:47 PM, Keith Wiley wrote:
> On Jan 28, 2011, at 15:50 , Greg Roelofs wrote:
>
>> Does your .so depend on any other potentially thread-unsafe .so that other
>> (non-Hadoop) processes might be using? System libraries like zlib are safe
>> (else they wouldn't make very good system libraries), but maybe some other
>> research library or something? (That's a long shot, but I'm pretty much
>> grasping at straws here.)
>
> Yeah, I dunno. It's a very complicated system that hits all kinds of popular conventional libraries: boost, eigen, countless other things. I doubt of it is being access however. This is a dedicated cluster so if my task is the only one running, then it's only concurrent with the OS itself (and the JVM and Hadoop).
By chance, do you have jvm reuse turned on?
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Greg Roelofs <ro...@yahoo-inc.com>.
I wrote:
>> Btw, keep in mind that there are memory-related bugs that don't show up
>> until there's something big in memory that pushes the code in question
>> up into a region with different data patterns in it (most frequently zero
>> vs. non-zero, but others are possible). IOW, maybe the code is dependent
>> on uninitialized memory, but you were getting lucky when you ran it outside
>> of Hadoop. Have you run it through valgrind or Purify or similar?
Keith Wiley wrote:
> Valgrind has turned out to be almost useless. It can't "reach"
> through the JVM through JNI to the .so code. If I don't
> tell valgrind to following children, it obviously produces
> no relevant output, but if I do tell it to follow children,
> it can't successfully launch a VM to run Java in:
> Error occurred during initialization of VM
> Unknown x64 processor: SSE2 not supported
> Sigh...any thoughts on running Valgrind on Hadoop->JVM->JNI->native code?
I actually meant something simpler: if we posit that the bug is actually
in the library code but isn't always triggering a segfault due to random
memory conditions (i.e., "getting lucky"), then running valgrind on it in
a non-Java context (i.e., what you said "runs perfectly fine outside Hadoop")
should detect such bug(s).
If that shows nothing, and you're not passing buffers across the JNI boundary
(=> possible GC issues, perhaps subtle ones?), then I'm out of ideas. Again.
Sorry. :-/
Greg
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Keith Wiley <kw...@keithwiley.com>.
On Jan 28, 2011, at 15:50 , Greg Roelofs wrote:
> Does your .so depend on any other potentially thread-unsafe .so that other
> (non-Hadoop) processes might be using? System libraries like zlib are safe
> (else they wouldn't make very good system libraries), but maybe some other
> research library or something? (That's a long shot, but I'm pretty much
> grasping at straws here.)
Yeah, I dunno. It's a very complicated system that hits all kinds of popular conventional libraries: boost, eigen, countless other things. I doubt of it is being access however. This is a dedicated cluster so if my task is the only one running, then it's only concurrent with the OS itself (and the JVM and Hadoop).
>> Yes, not thread safe, but what difference could that make if I
>> don't use the library in a multi-threaded fashion. One map task,
>> one node, one Java thread calling JNI and using the native code?
>> How do thread safety issues factor into this? I admit, it's
>> my theory that threads might be involved somehow, but I don't
>> understand how, I'm just shooting in the dark since I can't
>> solve this problem any other way yet.
>
> Since you can reproduce it in standalone mode, can you enable core dumps
> so you can see the backtrace of the code that segfaults? Knowing what
> specifically broke and how it got there is always a big help.
Yep, I've got core dumps and I've run them through gdb. I know that the code often dies very deep inside ostensibly standard libraries, like eigen for example...which leads me to believe the memory corruption happened long before the code reached that point.
> Btw, keep in mind that there are memory-related bugs that don't show up
> until there's something big in memory that pushes the code in question
> up into a region with different data patterns in it (most frequently zero
> vs. non-zero, but others are possible). IOW, maybe the code is dependent
> on uninitialized memory, but you were getting lucky when you ran it outside
> of Hadoop. Have you run it through valgrind or Purify or similar?
Valgrind has turned out to be almost useless. It can't "reach" through the JVM through JNI to the .so code. If I don't tell valgrind to following children, it obviously produces no relevant output, but if I do tell it to follow children, it can't successfully launch a VM to run Java in:
Error occurred during initialization of VM
Unknown x64 processor: SSE2 not supported
Sigh...any thoughts on running Valgrind on Hadoop->JVM->JNI->native code?
Thanks.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"Yet mark his perfect self-contentment, and hence learn his lesson, that to be
self-contented is to be vile and ignorant, and that to aspire is better than to
be blindly and impotently happy."
-- Edwin A. Abbott, Flatland
________________________________________________________________________________
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Greg Roelofs <ro...@yahoo-inc.com>.
Keith Wiley wrote:
> (1) Speculative execution would occur on a completely different
> node, so there definitely isn't any thread cross-talk (in memory).
> So long as they don't rely on reading/writing temp files from
> HDFS I don't see how they could have any effect on one another.
Good point.
> (2) I am also getting seg faults when I run in noncluster
> standalone mode, which is a single nonspeculated thread......I
> presume.
That's the same as "pseudo-distributed mode"?
> Can you explain your thoughts on speculative execution w.r.t. the
> problems I'm having?
Thoughts? You expect me to have thoughts, too??
:-)
I had not fully thought through the spec ex idea; it was the only thing
I could think of that might put two (otherwise independent) JNI-using tasks
onto the same node. But as you point out above, it wouldn't...
Does your .so depend on any other potentially thread-unsafe .so that other
(non-Hadoop) processes might be using? System libraries like zlib are safe
(else they wouldn't make very good system libraries), but maybe some other
research library or something? (That's a long shot, but I'm pretty much
grasping at straws here.)
> Yes, not thread safe, but what difference could that make if I
> don't use the library in a multi-threaded fashion. One map task,
> one node, one Java thread calling JNI and using the native code?
> How do thread safety issues factor into this? I admit, it's
> my theory that threads might be involved somehow, but I don't
> understand how, I'm just shooting in the dark since I can't
> solve this problem any other way yet.
Since you can reproduce it in standalone mode, can you enable core dumps
so you can see the backtrace of the code that segfaults? Knowing what
specifically broke and how it got there is always a big help.
Btw, keep in mind that there are memory-related bugs that don't show up
until there's something big in memory that pushes the code in question
up into a region with different data patterns in it (most frequently zero
vs. non-zero, but others are possible). IOW, maybe the code is dependent
on uninitialized memory, but you were getting lucky when you ran it outside
of Hadoop. Have you run it through valgrind or Purify or similar?
Greg
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Keith Wiley <kw...@keithwiley.com>.
On Jan 28, 2011, at 13:37 , Greg Roelofs wrote:
> Keith Wiley wrote:
>
>> However, it is also not run in any concurrent fashion except
>> w.r.t. to the JVM itself. For example, my map task doesn't
>> make parallel calls through JNI to the native code on concurrent
>> threads at the Java level, nor does the native code itself spawn
>> any threads (like I said, it isn't even compiled with pthreads).
>
> Is speculative execution enabled?
I suppose I hadn't specifically disabled that, but two points:
(1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory). So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another.
(2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume.
Can you explain your thoughts on speculative execution w.r.t. the problems I'm having?
>> So, the question is, in the scenario I have described, is there
>> any reason to suspect that the cause of my problems is some
>> sort of thread trampling between the native code and something
>> else in the surrounding environment (the JVM or something like
>> that), especially in the context of the surrounding Hadoop
>> infrastructure? It doesn't really make any sense to me, but
>> I'm running out of ideas.
>
> I don't see any obvious possibilities except speculative execution, and
> even that would depend on how the shared library was written. Does it
> contain any global or static variables? If so, it's almost certainly
> not thread-safe (unless, say, a global variable were basically write-
> only and advisory only, e.g., used only in an error message or a summary
> message at the end).
Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion. One map task, one node, one Java thread calling JNI and using the native code? How do thread safety issues factor into this? I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet.
Thanks for the input. Can you tell me what you're thinking w.r.t. speculative execution?
I'll try it without, but I don't see how it could alter the standalone behavior.
Cheers!
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"Luminous beings are we, not this crude matter."
-- Yoda
________________________________________________________________________________
Re: Thread safety issues with JNI/native code from map tasks?
Posted by Greg Roelofs <ro...@yahoo-inc.com>.
Keith Wiley wrote:
> However, it is also not run in any concurrent fashion except
> w.r.t. to the JVM itself. For example, my map task doesn't
> make parallel calls through JNI to the native code on concurrent
> threads at the Java level, nor does the native code itself spawn
> any threads (like I said, it isn't even compiled with pthreads).
Is speculative execution enabled?
> So, the question is, in the scenario I have described, is there
> any reason to suspect that the cause of my problems is some
> sort of thread trampling between the native code and something
> else in the surrounding environment (the JVM or something like
> that), especially in the context of the surrounding Hadoop
> infrastructure? It doesn't really make any sense to me, but
> I'm running out of ideas.
I don't see any obvious possibilities except speculative execution, and
even that would depend on how the shared library was written. Does it
contain any global or static variables? If so, it's almost certainly
not thread-safe (unless, say, a global variable were basically write-
only and advisory only, e.g., used only in an error message or a summary
message at the end).
No other ideas.
Greg