You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by "Geir Magnusson Jr." <ge...@pobox.com> on 2007/01/09 14:07:51 UTC
[drlvm] stress.Mix / MegaSpawn threading bug
I started a new thread because I think this is really important.
I've also added a page in the wiki to track this stuff, because I
can't keep it in my head:
http://wiki.apache.org/harmony/MegaSpawnThreadingBug
which you can get to from the home page via the "WhiteBoards"
section, intended to be a place where we can work as a team on a
whiteboard, with the intention that once the mini-project is over, we
erase...
I think this is a scary scary problem :)
geir
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 12:19 AM, Aleksey Ignatenko wrote:
> On 1/10/07, Rana Dasgupta <rd...@gmail.com> wrote:
>>
>> On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
>> >
>> > On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
>> > >> I've tried to analyze MegaSpawn test on windows and here's
>> what I
>> found
>> > >> out.
>> > >>
>> > >> OOME is thrown because process virtual size easily gets up to
>> 2Gb.
>> This
>> > >> happens at about ~1.5k simultaneously running threads. I
>> think it
>> > >> happens because all of virtual process memory is mapped for
>> thread
>> > stacks.
>> > >>
>> >
>> > >Good job! I got the same sort of hunch when I looked at the
>> source
>> code
>> > did
>> > >not have enough time to pin down specifics. The only guidance I
>> > >found regarding what happens when too many threads are spawned
>> is the
>> > >following in the java.lang.Thread reference manual,
>> "...specifying a
>> > lower
>> > >[stacksize] value may allow a greater number of threads to exist
>> > >concurrently without throwing an OutOfMemoryError (or other
>> internal
>> > >error)."
>> >
>> > >I think what the above implies is that it is OK for the JVM to
>> error
>> and
>> > >exit if the app tries to create too many threads. If this is
>> the case,
>> > it
>> > >sort of looks like we need to clean up the handling of malloc()
>> errors
>> so
>> > >that the JVM can exit gracefully.
>>
>>
>> I am not sure that we need to do something about this. The default
>> initial
>> stack size on Windows is 1M, and that is the recommended init size
>> for
>> real
>> applications. The fact that our threads start with a larger intial
>> stack
>> mapped( default ) than RI is a design issue, it is not a bug. We
>> could
>> start
>> with 2K and create many more threads! Exactly as Gregory points out,
>> ultimately we will hit virtual memory limits and fail. The reason
>> the RI
>> seems to fail less is that the test ends before running out of
>> virtual
>> memory.On my 32 bit RHEL Linux box, RI fails almost every time with
>> MegaSpawn, with an identical OOME error message and stack dump.
>>
>> We can catch the exception in the test and print a message. But I
>> am not
>> very sure what purpose that would serve. A resource exhaustion
>> exception
>> is
>> a fatal exception and the process is hosed, no real app would be
>> able to
>> do
>> anything more at this point. We should not use this test ( which
>> is not a
>> real app ) as guidance to tune the initial stack size. My
>> suggestion is to
>> lower the test duration so that we can create about a 1000( or
>> whatever
>> magic number ) threads at least. That is the stress condition we
>> should
>> test
>> for.
>
>
> Can we find such a "magic number" for test? The test is highly
> dependent on
> PC configuration (size of available virtual memory), so someone
> with 256Mb
> installed only could have OOME problems with lower number of
> threads. Looks
> like the root problem is in incorrect handling of native memory
> exhaustion.
<b>+1</b>
geir
>
> Best regards,
> Aleksey.
>
> Thanks,
>> Rana
>>
>>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Weldon Washburn <we...@gmail.com>.
On 1/9/07, Aleksey Ignatenko <al...@gmail.com> wrote:
>
> On 1/10/07, Rana Dasgupta <rd...@gmail.com> wrote:
> >
> > On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
> > >
> > > On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
> > > >> I've tried to analyze MegaSpawn test on windows and here's what I
> > found
> > > >> out.
> > > >>
> > > >> OOME is thrown because process virtual size easily gets up to 2Gb.
> > This
> > > >> happens at about ~1.5k simultaneously running threads. I think it
> > > >> happens because all of virtual process memory is mapped for thread
> > > stacks.
> > > >>
> > >
> > > >Good job! I got the same sort of hunch when I looked at the source
> > code
> > > did
> > > >not have enough time to pin down specifics. The only guidance I
> > > >found regarding what happens when too many threads are spawned is the
> > > >following in the java.lang.Thread reference manual, "...specifying a
> > > lower
> > > >[stacksize] value may allow a greater number of threads to exist
> > > >concurrently without throwing an OutOfMemoryError (or other internal
> > > >error)."
> > >
> > > >I think what the above implies is that it is OK for the JVM to error
> > and
> > > >exit if the app tries to create too many threads. If this is the
> case,
> > > it
> > > >sort of looks like we need to clean up the handling of malloc()
> errors
> > so
> > > >that the JVM can exit gracefully.
> >
> >
> > I am not sure that we need to do something about this. The default
> initial
> > stack size on Windows is 1M, and that is the recommended init size for
> > real
> > applications. The fact that our threads start with a larger intial stack
> > mapped( default ) than RI is a design issue, it is not a bug. We could
> > start
> > with 2K and create many more threads! Exactly as Gregory points out,
> > ultimately we will hit virtual memory limits and fail. The reason the RI
> > seems to fail less is that the test ends before running out of virtual
> > memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> > MegaSpawn, with an identical OOME error message and stack dump.
> >
> > We can catch the exception in the test and print a message. But I am not
> > very sure what purpose that would serve. A resource exhaustion exception
> > is
> > a fatal exception and the process is hosed, no real app would be able to
> > do
> > anything more at this point. We should not use this test ( which is not
> a
> > real app ) as guidance to tune the initial stack size. My suggestion is
> to
> > lower the test duration so that we can create about a 1000( or whatever
> > magic number ) threads at least. That is the stress condition we should
> > test
> > for.
>
>
> Can we find such a "magic number" for test? The test is highly dependent
> on
> PC configuration (size of available virtual memory), so someone with 256Mb
> installed only could have OOME problems with lower number of threads.
I agree.
Looks
> like the root problem is in incorrect handling of native memory
> exhaustion.
I agree. I put stress.Mix on the exclusion list for now. And committed the
modified exclusion list.
Best regards,
> Aleksey.
>
> Thanks,
> > Rana
> >
> >
>
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
Aleksey,
We are hitting limits in virtual memory, not in the amount of RAM the
machine has.
Rana
On 1/9/07, Aleksey Ignatenko <al...@gmail.com> wrote:
>
>
> >Can we find such a "magic number" for test? The test is highly dependent
> on
> >PC configuration (size of available virtual memory), so someone with
> 256Mb
> >installed only could have OOME problems with lower number of threads.
> Looks
> >like the root problem is in incorrect handling of native memory
> exhaustion.
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Aleksey Ignatenko <al...@gmail.com>.
On 1/10/07, Rana Dasgupta <rd...@gmail.com> wrote:
>
> On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
> >
> > On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
> > >> I've tried to analyze MegaSpawn test on windows and here's what I
> found
> > >> out.
> > >>
> > >> OOME is thrown because process virtual size easily gets up to 2Gb.
> This
> > >> happens at about ~1.5k simultaneously running threads. I think it
> > >> happens because all of virtual process memory is mapped for thread
> > stacks.
> > >>
> >
> > >Good job! I got the same sort of hunch when I looked at the source
> code
> > did
> > >not have enough time to pin down specifics. The only guidance I
> > >found regarding what happens when too many threads are spawned is the
> > >following in the java.lang.Thread reference manual, "...specifying a
> > lower
> > >[stacksize] value may allow a greater number of threads to exist
> > >concurrently without throwing an OutOfMemoryError (or other internal
> > >error)."
> >
> > >I think what the above implies is that it is OK for the JVM to error
> and
> > >exit if the app tries to create too many threads. If this is the case,
> > it
> > >sort of looks like we need to clean up the handling of malloc() errors
> so
> > >that the JVM can exit gracefully.
>
>
> I am not sure that we need to do something about this. The default initial
> stack size on Windows is 1M, and that is the recommended init size for
> real
> applications. The fact that our threads start with a larger intial stack
> mapped( default ) than RI is a design issue, it is not a bug. We could
> start
> with 2K and create many more threads! Exactly as Gregory points out,
> ultimately we will hit virtual memory limits and fail. The reason the RI
> seems to fail less is that the test ends before running out of virtual
> memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> MegaSpawn, with an identical OOME error message and stack dump.
>
> We can catch the exception in the test and print a message. But I am not
> very sure what purpose that would serve. A resource exhaustion exception
> is
> a fatal exception and the process is hosed, no real app would be able to
> do
> anything more at this point. We should not use this test ( which is not a
> real app ) as guidance to tune the initial stack size. My suggestion is to
> lower the test duration so that we can create about a 1000( or whatever
> magic number ) threads at least. That is the stress condition we should
> test
> for.
Can we find such a "magic number" for test? The test is highly dependent on
PC configuration (size of available virtual memory), so someone with 256Mb
installed only could have OOME problems with lower number of threads. Looks
like the root problem is in incorrect handling of native memory exhaustion.
Best regards,
Aleksey.
Thanks,
> Rana
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
Yes this makes sense. Starting out with a suitable initial stack size on
different platforms is reasonble. We also found that the way the thread
stack gets mapped on Windows and Linux is different ( Windows incrementally
maps with a guardpage, whereas Linux maps most of it upfront ). I am
surethat several such considerations go into choosing the init size for the
IBM VM. We need to tune the init stack sizes too, but this test does not
provide enough good data to set these values.
Thanks,
Rana
On 1/10/07, Tim Ellison <t....@gmail.com> wrote:
>
> Rana Dasgupta wrote:
> > I am not sure that we need to do something about this. The default
> initial
> > stack size on Windows is 1M, and that is the recommended init size for
> real
> > applications. The fact that our threads start with a larger intial stack
> > mapped( default ) than RI is a design issue, it is not a bug. We could
> > start
> > with 2K and create many more threads! Exactly as Gregory points out,
> > ultimately we will hit virtual memory limits and fail. The reason the RI
> > seems to fail less is that the test ends before running out of virtual
> > memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> > MegaSpawn, with an identical OOME error message and stack dump.
>
> FWIW the IBM VM 'design point' is to create threads on 32-bit Windows
> with an OS stack size of 32K for just this reason. On all other 32-bit
> platforms, and 64-bit Windows, the initial size is 256K.
>
> Regards,
> Tim
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 4:30 AM, Tim Ellison wrote:
> Rana Dasgupta wrote:
>> I am not sure that we need to do something about this. The default
>> initial
>> stack size on Windows is 1M, and that is the recommended init size
>> for real
>> applications. The fact that our threads start with a larger intial
>> stack
>> mapped( default ) than RI is a design issue, it is not a bug. We
>> could
>> start
>> with 2K and create many more threads! Exactly as Gregory points out,
>> ultimately we will hit virtual memory limits and fail. The reason
>> the RI
>> seems to fail less is that the test ends before running out of
>> virtual
>> memory.On my 32 bit RHEL Linux box, RI fails almost every time with
>> MegaSpawn, with an identical OOME error message and stack dump.
>
> FWIW the IBM VM 'design point' is to create threads on 32-bit Windows
> with an OS stack size of 32K for just this reason. On all other 32-
> bit
> platforms, and 64-bit Windows, the initial size is 256K.
Sounds like a reasonable starting point for us :)
Thanks - that's really useful info.
geir
>
> Regards,
> Tim
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Tim Ellison <t....@gmail.com>.
Rana Dasgupta wrote:
> I am not sure that we need to do something about this. The default initial
> stack size on Windows is 1M, and that is the recommended init size for real
> applications. The fact that our threads start with a larger intial stack
> mapped( default ) than RI is a design issue, it is not a bug. We could
> start
> with 2K and create many more threads! Exactly as Gregory points out,
> ultimately we will hit virtual memory limits and fail. The reason the RI
> seems to fail less is that the test ends before running out of virtual
> memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> MegaSpawn, with an identical OOME error message and stack dump.
FWIW the IBM VM 'design point' is to create threads on 32-bit Windows
with an OS stack size of 32K for just this reason. On all other 32-bit
platforms, and 64-bit Windows, the initial size is 256K.
Regards,
Tim
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 10:42 PM, Rana Dasgupta wrote:
> On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>>
>> On Jan 10, 2007, at 2:13 PM, Weldon Washburn wrote:
>>
>> >> 1)
>> >> In some earlier posting, it was mentioned that somehow the virtual
>> >> memory
>> >> address space is impacted by how much physical memory is in a
>> given
>> >> computer. Actually this is not true. The virtual address space
>> >> available
>> >> to the JVM is fixed by the OS. A machine with less phys mem will
>> >> do more
>> >> disk I/O. In other words "C" malloc() hard limits are set by OS
>> >>version
>> >> number not by RAM chips.
>> >>
>>
>> >Talking about VM vs RAM vs whatever is a red herring - we may be
>> >ported to a machine w/o virtual memory. What matters is that when
>> >malloc() returns null, we do something smart. At least, do nothing
>> >harmful.
>
>
> There can be no machine without virtual memory on any of the OS's of
> interest to us.
Who do you mean by "us"? I'm not included in that "us". Sure, the
most popular and probable ports of harmony will be to environments
with virtual memory, but you never know what people will want to do
with it.
Anyway, dealing with resource limits is simply good programming
practice, even if you have virtual memory :)
> VM is not a type of memory technology. What Weldon, Gregory
> and several others have pointed out is that if one keeps on consuming
> virtual address space by allocating space for thread stacks, the
> address
> space will eventually run out, and the process will be a fatal state
> independent of what is the physical memory on the machine.
I don't think that anyone is arguing with this point. But from a
coding POV, it's utterly irrelevant. We seem to have a bit of
sloppyness deep down in DRLVM, and we need to fix it. End of story.
>
>>> 2)
>> >> Why not simply hard code DRLVM to throw an OOME whenever there are
>> >> more than
>> >> 1K threads running? I think Rana first suggested this approach.
>> >> My guess
>> >> is that 1K threads is good enough to run lots of interesting
>> >> workloads. My
>> >> guess is that common versions of WinXP and Linux will handle the C
>> >> malloc()
>> >> load of 1K threads successfully. If not, how about trying 512
>> >> threads?
>>
>> >Because this is picking up the rug, and sweeping all the dirt
>> >underneath it. The core problem isn't that we try too many threads,
>> >but the code wasn't written defensively. Putting an artificial
>> limit
>> >on # of threads just means that we'll hit it somewhere else, in some
>> >other resource usage.
>>
>> >I think we should fix it.
>
>
> Sure. The way to fix a fatal error is to leave room for a process
> to recover
> from it or handle it. Another example of a fatal error is a Stack
> overflow
> or a TerminateProcess signal. In the case of Stack overflow, we
> handle it by
> trying to raise the exception while some room is left of the stack
> so that
> there is a fair chance to handle. Similarly, an approach could be
> to set a
> limit on the maximum number of threads we create. Based on the
> memory we
> give each thread stack we can choose a limit which we estimate will
> leave us
> room to handle the error.
Ok, so we do actually agree.
>
>>> There seem to be some basic things we can do, like reduce the stack
>> >>size on windows from the terabyte or whatever it is now, to the
>> >>number that our dear, esteemed colleague from IBM claims is
>> perfectly
>> >>suitable for production use.
>>
>> >That too doesn't solve the problem, but it certainly fixes a problem
>> >we are now aware of - our stack size is too big.... :)
>
>
> The best size to set for the thread stack is a valid issue, and it
> is useful
> information to know what the IBM VM sets. Google searches also seem
> to show
> that threadstack size on J9 is user configurable. But even with
> smaller
> stack sizes, if one ran Megaspawn for sufficiently long time, we
> would get
> the same error. So we can't have unbounded stresses like this or,
> the VM
> needs to bound the resources consumable by such a test. Also, we
> cannot just
> emulate what the IBM VM does in one specific area without
> understanding all
> their entire design. For example, a small stack size will cause Stack
> Overflow exceptions to happen early. We need to tune these sizes
> based on
> our own experiments.
Yep
>
>>>
>> >> 3)
>> >> The above does not deal with the general architecture question of
>> >> handling C
>> >> malloc failures. This is far harder to solve. Note that solving
>> >> the big
>> >> question will also require far more extensive regression tests
>> than
>> >> MegaSpawn. However, it does fix DRLVM so that it does not crash/
>> >> burn on
>> >> threads overload. This, in turn, gives us time to fix the real
>> >> underlying
>> >> problem(s) with C malloc.
>
>
> I think that we should defer this part, it is a dificult problem
> and there
> are several potential approaches based on what kind of reliable
> computing
> contracts we want to expose. For example, one can think of a contract
> that no fatal failures( OOME, stack overflow, thread abort ) happen
> in marked regions of code, ever. I don't think that we need to
> solve this
> hard problem right now.
I disagree. I'd prefer that we at least investigate why we handle
this situation so badly, and at least get to a point where the
situation was handled consistently.
geir
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 11, 2007, at 3:08 PM, Gregory Shimansky wrote:
> Weldon Washburn wrote:
>> On 1/11/07, Salikh Zakirov <Sa...@intel.com> wrote:
>>>
>>>
>>> >> >> 2)
>>> >> > >> Why not simply hard code DRLVM to throw an OOME whenever
>>> there are
>>> >> > >> more than
>>> >> > >> 1K threads running? I think Rana first suggested this
>>> approach.
>>> >> > >> My guess
>>> >> > >> is that 1K threads is good enough to run lots of interesting
>>> >> > >> workloads. My
>>> >> > >> guess is that common versions of WinXP and Linux will
>>> handle the C
>>> >> > >> malloc()
>>> >> > >> load of 1K threads successfully. If not, how about
>>> trying 512
>>> >> > >> threads?
>>>
>>> I've tried a quick-and-dirty patch to limit the number of threads
>>> to 500,
>>> and it indeed fixes crashes on MegaSpawn.
>> This is good news. Thanks for trying this out. One question.
>> Does DRLVM
>> handle OOME properly and gracefully when threads are limited to 500?
>
> If you look in another message in this thread, the investigation
> shows that breaks the test. It if the fact that there is *really*
> no memory left, and other allocations fail. This leads to crashes
> and hangs.
>
> Limiting threads number in this case creates a false OOME for
> Thread.start because actually there is more virtual memory
> available, and so allocating it is possible. In this case all
> you'll see is many OOME stack traces, but the test won't crash on
> hang.
>
> This is not a valid solution however because hitting memory limit
> may happen with 500 just the same if you configure OS to create
> threads with default stack of 100Mb, or it can happen because there
> is no available memory because of another reason.
>
> I would very much like if we fix this stress test correctly and do
> one step towards the right handing of OOME condition.
+1
Also, note that talking about virtual memory is pointless. Malloc
fails. it doesn't matter if there is virtual memory or not - we're
out of memory.
I'm impressed how fast so much clear analysis happened already...
geir
>
> --
> Gregory
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Weldon Washburn wrote:
> On 1/11/07, Salikh Zakirov <Sa...@intel.com> wrote:
>>
>>
>> >> >> 2)
>> >> > >> Why not simply hard code DRLVM to throw an OOME whenever there
>> are
>> >> > >> more than
>> >> > >> 1K threads running? I think Rana first suggested this approach.
>> >> > >> My guess
>> >> > >> is that 1K threads is good enough to run lots of interesting
>> >> > >> workloads. My
>> >> > >> guess is that common versions of WinXP and Linux will handle
>> the C
>> >> > >> malloc()
>> >> > >> load of 1K threads successfully. If not, how about trying 512
>> >> > >> threads?
>>
>> I've tried a quick-and-dirty patch to limit the number of threads to 500,
>> and it indeed fixes crashes on MegaSpawn.
>
>
> This is good news. Thanks for trying this out. One question. Does DRLVM
> handle OOME properly and gracefully when threads are limited to 500?
If you look in another message in this thread, the investigation shows
that breaks the test. It if the fact that there is *really* no memory
left, and other allocations fail. This leads to crashes and hangs.
Limiting threads number in this case creates a false OOME for
Thread.start because actually there is more virtual memory available,
and so allocating it is possible. In this case all you'll see is many
OOME stack traces, but the test won't crash on hang.
This is not a valid solution however because hitting memory limit may
happen with 500 just the same if you configure OS to create threads with
default stack of 100Mb, or it can happen because there is no available
memory because of another reason.
I would very much like if we fix this stress test correctly and do one
step towards the right handing of OOME condition.
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Weldon Washburn <we...@gmail.com>.
On 1/11/07, Salikh Zakirov <Sa...@intel.com> wrote:
>
>
> >> >> 2)
> >> > >> Why not simply hard code DRLVM to throw an OOME whenever there are
> >> > >> more than
> >> > >> 1K threads running? I think Rana first suggested this approach.
> >> > >> My guess
> >> > >> is that 1K threads is good enough to run lots of interesting
> >> > >> workloads. My
> >> > >> guess is that common versions of WinXP and Linux will handle the C
> >> > >> malloc()
> >> > >> load of 1K threads successfully. If not, how about trying 512
> >> > >> threads?
>
> I've tried a quick-and-dirty patch to limit the number of threads to 500,
> and it indeed fixes crashes on MegaSpawn.
This is good news. Thanks for trying this out. One question. Does DRLVM
handle OOME properly and gracefully when threads are limited to 500?
> -1 for hard coding max number of threads.
>
> Alexey's proposal to watch virtual memory availability looks feasible,
> and will let us avoid hardcoding any number.
>
> My observation is that limiting number of concurrently running threads
> is by far easier than implementing some sort of custom memory manager
> with reserve areas etc, and will allow us to get over MegaSpawn failures.
One pitfall to avoid is second guessing how malloc() behaves when there is
limited virtual memory available. In specific, malloc() may want to grab
50MB of virtual space but the OS only has eighteen 45MB chunks or smaller
available. And, of course, malloc/OS behavior varies from platform to
platform. A simpler, easier to debug, maintain approach for right now would
be to set a hard limit.
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Alexey Varlamov <al...@gmail.com>.
2007/1/12, Salikh Zakirov <Sa...@intel.com>:
>
> >> >> 2)
> >> > >> Why not simply hard code DRLVM to throw an OOME whenever there are
> >> > >> more than
> >> > >> 1K threads running? I think Rana first suggested this approach.
> >> > >> My guess
> >> > >> is that 1K threads is good enough to run lots of interesting
> >> > >> workloads. My
> >> > >> guess is that common versions of WinXP and Linux will handle the C
> >> > >> malloc()
> >> > >> load of 1K threads successfully. If not, how about trying 512
> >> > >> threads?
>
> I've tried a quick-and-dirty patch to limit the number of threads to 500,
> and it indeed fixes crashes on MegaSpawn.
>
> > -1 for hard coding max number of threads.
>
> Alexey's proposal to watch virtual memory availability looks feasible,
> and will let us avoid hardcoding any number.
Yes, but it will not work until exact key points for memory check-up
are specified.
And probably finding all such points is nearly equivalent to fixing
them for error conditions :)
>
> My observation is that limiting number of concurrently running threads
> is by far easier than implementing some sort of custom memory manager
> with reserve areas etc, and will allow us to get over MegaSpawn failures.
>
But, as many already pointed, what is the benefit of rounding
particular corner cases? It does not really help gaining better
reliability etc.
The easiest solution is just to put MegaSpawn in exclude list
forever^H^H^H^H^H^H^H until its' time comes ;)
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Salikh Zakirov <Sa...@Intel.com>.
>> >> 2)
>> > >> Why not simply hard code DRLVM to throw an OOME whenever there are
>> > >> more than
>> > >> 1K threads running? I think Rana first suggested this approach.
>> > >> My guess
>> > >> is that 1K threads is good enough to run lots of interesting
>> > >> workloads. My
>> > >> guess is that common versions of WinXP and Linux will handle the C
>> > >> malloc()
>> > >> load of 1K threads successfully. If not, how about trying 512
>> > >> threads?
I've tried a quick-and-dirty patch to limit the number of threads to 500,
and it indeed fixes crashes on MegaSpawn.
> -1 for hard coding max number of threads.
Alexey's proposal to watch virtual memory availability looks feasible,
and will let us avoid hardcoding any number.
My observation is that limiting number of concurrently running threads
is by far easier than implementing some sort of custom memory manager
with reserve areas etc, and will allow us to get over MegaSpawn failures.
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Evgueni Brevnov <ev...@gmail.com>.
On 1/11/07, Rana Dasgupta <rd...@gmail.com> wrote:
> On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >
> >
> > On Jan 10, 2007, at 2:13 PM, Weldon Washburn wrote:
> >
> > >> 1)
> > >> In some earlier posting, it was mentioned that somehow the virtual
> > >> memory
> > >> address space is impacted by how much physical memory is in a given
> > >> computer. Actually this is not true. The virtual address space
> > >> available
> > >> to the JVM is fixed by the OS. A machine with less phys mem will
> > >> do more
> > >> disk I/O. In other words "C" malloc() hard limits are set by OS
> > >>version
> > >> number not by RAM chips.
> > >>
> >
> > >Talking about VM vs RAM vs whatever is a red herring - we may be
> > >ported to a machine w/o virtual memory. What matters is that when
> > >malloc() returns null, we do something smart. At least, do nothing
> > >harmful.
>
>
> There can be no machine without virtual memory on any of the OS's of
> interest to us. VM is not a type of memory technology. What Weldon, Gregory
> and several others have pointed out is that if one keeps on consuming
> virtual address space by allocating space for thread stacks, the address
> space will eventually run out, and the process will be a fatal state
> independent of what is the physical memory on the machine.
>
> >> 2)
> > >> Why not simply hard code DRLVM to throw an OOME whenever there are
> > >> more than
> > >> 1K threads running? I think Rana first suggested this approach.
> > >> My guess
> > >> is that 1K threads is good enough to run lots of interesting
> > >> workloads. My
> > >> guess is that common versions of WinXP and Linux will handle the C
> > >> malloc()
> > >> load of 1K threads successfully. If not, how about trying 512
> > >> threads?
> >
-1 for hard coding max number of threads.
> > >Because this is picking up the rug, and sweeping all the dirt
> > >underneath it. The core problem isn't that we try too many threads,
> > >but the code wasn't written defensively. Putting an artificial limit
> > >on # of threads just means that we'll hit it somewhere else, in some
> > >other resource usage.
> >
> > >I think we should fix it.
>
>
> Sure. The way to fix a fatal error is to leave room for a process to recover
> from it or handle it. Another example of a fatal error is a Stack overflow
> or a TerminateProcess signal. In the case of Stack overflow, we handle it by
> trying to raise the exception while some room is left of the stack so that
> there is a fair chance to handle. Similarly, an approach could be to set a
> limit on the maximum number of threads we create. Based on the memory we
> give each thread stack we can choose a limit which we estimate will leave us
> room to handle the error.
>
> >>There seem to be some basic things we can do, like reduce the stack
> > >>size on windows from the terabyte or whatever it is now, to the
> > >>number that our dear, esteemed colleague from IBM claims is perfectly
> > >>suitable for production use.
> >
> > >That too doesn't solve the problem, but it certainly fixes a problem
> > >we are now aware of - our stack size is too big.... :)
>
>
> The best size to set for the thread stack is a valid issue, and it is useful
> information to know what the IBM VM sets. Google searches also seem to show
> that threadstack size on J9 is user configurable. But even with smaller
> stack sizes, if one ran Megaspawn for sufficiently long time, we would get
> the same error. So we can't have unbounded stresses like this or, the VM
> needs to bound the resources consumable by such a test. Also, we cannot just
> emulate what the IBM VM does in one specific area without understanding all
> their entire design. For example, a small stack size will cause Stack
> Overflow exceptions to happen early. We need to tune these sizes based on
> our own experiments.
>
> >>
> > >> 3)
> > >> The above does not deal with the general architecture question of
> > >> handling C
> > >> malloc failures. This is far harder to solve. Note that solving
> > >> the big
> > >> question will also require far more extensive regression tests than
> > >> MegaSpawn. However, it does fix DRLVM so that it does not crash/
> > >> burn on
> > >> threads overload. This, in turn, gives us time to fix the real
> > >> underlying
> > >> problem(s) with C malloc.
>
>
> I think that we should defer this part, it is a dificult problem and there
> are several potential approaches based on what kind of reliable computing
> contracts we want to expose. For example, one can think of a contract
> that no fatal failures( OOME, stack overflow, thread abort ) happen
> in marked regions of code, ever. I don't think that we need to solve this
> hard problem right now.
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
> On Jan 10, 2007, at 2:13 PM, Weldon Washburn wrote:
>
> >> 1)
> >> In some earlier posting, it was mentioned that somehow the virtual
> >> memory
> >> address space is impacted by how much physical memory is in a given
> >> computer. Actually this is not true. The virtual address space
> >> available
> >> to the JVM is fixed by the OS. A machine with less phys mem will
> >> do more
> >> disk I/O. In other words "C" malloc() hard limits are set by OS
> >>version
> >> number not by RAM chips.
> >>
>
> >Talking about VM vs RAM vs whatever is a red herring - we may be
> >ported to a machine w/o virtual memory. What matters is that when
> >malloc() returns null, we do something smart. At least, do nothing
> >harmful.
There can be no machine without virtual memory on any of the OS's of
interest to us. VM is not a type of memory technology. What Weldon, Gregory
and several others have pointed out is that if one keeps on consuming
virtual address space by allocating space for thread stacks, the address
space will eventually run out, and the process will be a fatal state
independent of what is the physical memory on the machine.
>> 2)
> >> Why not simply hard code DRLVM to throw an OOME whenever there are
> >> more than
> >> 1K threads running? I think Rana first suggested this approach.
> >> My guess
> >> is that 1K threads is good enough to run lots of interesting
> >> workloads. My
> >> guess is that common versions of WinXP and Linux will handle the C
> >> malloc()
> >> load of 1K threads successfully. If not, how about trying 512
> >> threads?
>
> >Because this is picking up the rug, and sweeping all the dirt
> >underneath it. The core problem isn't that we try too many threads,
> >but the code wasn't written defensively. Putting an artificial limit
> >on # of threads just means that we'll hit it somewhere else, in some
> >other resource usage.
>
> >I think we should fix it.
Sure. The way to fix a fatal error is to leave room for a process to recover
from it or handle it. Another example of a fatal error is a Stack overflow
or a TerminateProcess signal. In the case of Stack overflow, we handle it by
trying to raise the exception while some room is left of the stack so that
there is a fair chance to handle. Similarly, an approach could be to set a
limit on the maximum number of threads we create. Based on the memory we
give each thread stack we can choose a limit which we estimate will leave us
room to handle the error.
>>There seem to be some basic things we can do, like reduce the stack
> >>size on windows from the terabyte or whatever it is now, to the
> >>number that our dear, esteemed colleague from IBM claims is perfectly
> >>suitable for production use.
>
> >That too doesn't solve the problem, but it certainly fixes a problem
> >we are now aware of - our stack size is too big.... :)
The best size to set for the thread stack is a valid issue, and it is useful
information to know what the IBM VM sets. Google searches also seem to show
that threadstack size on J9 is user configurable. But even with smaller
stack sizes, if one ran Megaspawn for sufficiently long time, we would get
the same error. So we can't have unbounded stresses like this or, the VM
needs to bound the resources consumable by such a test. Also, we cannot just
emulate what the IBM VM does in one specific area without understanding all
their entire design. For example, a small stack size will cause Stack
Overflow exceptions to happen early. We need to tune these sizes based on
our own experiments.
>>
> >> 3)
> >> The above does not deal with the general architecture question of
> >> handling C
> >> malloc failures. This is far harder to solve. Note that solving
> >> the big
> >> question will also require far more extensive regression tests than
> >> MegaSpawn. However, it does fix DRLVM so that it does not crash/
> >> burn on
> >> threads overload. This, in turn, gives us time to fix the real
> >> underlying
> >> problem(s) with C malloc.
I think that we should defer this part, it is a dificult problem and there
are several potential approaches based on what kind of reliable computing
contracts we want to expose. For example, one can think of a contract
that no fatal failures( OOME, stack overflow, thread abort ) happen
in marked regions of code, ever. I don't think that we need to solve this
hard problem right now.
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 2:13 PM, Weldon Washburn wrote:
> Some observations and a simple patch that might just work well
> enough for
> the current state of DRLVM development.
>
> 1)
> In some earlier posting, it was mentioned that somehow the virtual
> memory
> address space is impacted by how much physical memory is in a given
> computer. Actually this is not true. The virtual address space
> available
> to the JVM is fixed by the OS. A machine with less phys mem will
> do more
> disk I/O. In other words "C" malloc() hard limits are set by OS
> version
> number not by RAM chips.
>
Talking about VM vs RAM vs whatever is a red herring - we may be
ported to a machine w/o virtual memory. What matters is that when
malloc() returns null, we do something smart. At least, do nothing
harmful.
> 2)
> Why not simply hard code DRLVM to throw an OOME whenever there are
> more than
> 1K threads running? I think Rana first suggested this approach.
> My guess
> is that 1K threads is good enough to run lots of interesting
> workloads. My
> guess is that common versions of WinXP and Linux will handle the C
> malloc()
> load of 1K threads successfully. If not, how about trying 512
> threads?
Because this is picking up the rug, and sweeping all the dirt
underneath it. The core problem isn't that we try too many threads,
but the code wasn't written defensively. Putting an artificial limit
on # of threads just means that we'll hit it somewhere else, in some
other resource usage.
I think we should fix it.
There seem to be some basic things we can do, like reduce the stack
size on windows from the terabyte or whatever it is now, to the
number that our dear, esteemed colleague from IBM claims is perfectly
suitable for production use.
That too doesn't solve the problem, but it certainly fixes a problem
we are now aware of - our stack size is too big.... :)
>
> 3)
> The above does not deal with the general architecture question of
> handling C
> malloc failures. This is far harder to solve. Note that solving
> the big
> question will also require far more extensive regression tests than
> MegaSpawn. However, it does fix DRLVM so that it does not crash/
> burn on
> threads overload. This, in turn, gives us time to fix the real
> underlying
> problem(s) with C malloc.
From this perspective, I don't mind as much, as long as we fix the
stack sizes. But a part of me wants to say no, because there's
nothing compelling us to actually fix the problem. :)
Maybe we set the number to something crippling, like 10 or something,
which will then motivate anyone who wants to o something useful with
the VM
I'm just always nervous about things like this...
geir
>
>
> On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>>
>> On Jan 10, 2007, at 8:51 AM, Gregory Shimansky wrote:
>>
>> > Geir Magnusson Jr. wrote:
>> >> The big thing for me is ensuring that we can drive the VM to the
>> >> limit, and it maintains internal integrity, so applications that
>> >> are designed to gracefully deal with resource exhaustion can do so
>> >> w/ confidence that the VM isn't about to crume out from
>> >> underneath them.
>> >
>> > I agree with Geir that we should try to handle out of C heap
>> > condition gracefully. The problem is that there is no clearly
>> > defined contract for many functions that use memory allocation
>> > about what to do in case of out of memory condition.
>> >
>> > To maintain integrity all VM functions which allocate memory from C
>> > heap should return gracefully all the way up the stack until they
>> > hit Java code that called them and then OOME exception shall be
>> > seen by the Java code. It is not an easy task because all code
>> > paths should support it, including JIT and GC.
>> >
>>
>> Agreed. But certainly worth striving for :)
>>
>> geir
>>
>>
>>
>
>
> --
> Weldon Washburn
> Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Weldon Washburn <we...@gmail.com>.
Some observations and a simple patch that might just work well enough for
the current state of DRLVM development.
1)
In some earlier posting, it was mentioned that somehow the virtual memory
address space is impacted by how much physical memory is in a given
computer. Actually this is not true. The virtual address space available
to the JVM is fixed by the OS. A machine with less phys mem will do more
disk I/O. In other words "C" malloc() hard limits are set by OS version
number not by RAM chips.
2)
Why not simply hard code DRLVM to throw an OOME whenever there are more than
1K threads running? I think Rana first suggested this approach. My guess
is that 1K threads is good enough to run lots of interesting workloads. My
guess is that common versions of WinXP and Linux will handle the C malloc()
load of 1K threads successfully. If not, how about trying 512 threads?
3)
The above does not deal with the general architecture question of handling C
malloc failures. This is far harder to solve. Note that solving the big
question will also require far more extensive regression tests than
MegaSpawn. However, it does fix DRLVM so that it does not crash/burn on
threads overload. This, in turn, gives us time to fix the real underlying
problem(s) with C malloc.
On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
> On Jan 10, 2007, at 8:51 AM, Gregory Shimansky wrote:
>
> > Geir Magnusson Jr. wrote:
> >> The big thing for me is ensuring that we can drive the VM to the
> >> limit, and it maintains internal integrity, so applications that
> >> are designed to gracefully deal with resource exhaustion can do so
> >> w/ confidence that the VM isn't about to crume out from
> >> underneath them.
> >
> > I agree with Geir that we should try to handle out of C heap
> > condition gracefully. The problem is that there is no clearly
> > defined contract for many functions that use memory allocation
> > about what to do in case of out of memory condition.
> >
> > To maintain integrity all VM functions which allocate memory from C
> > heap should return gracefully all the way up the stack until they
> > hit Java code that called them and then OOME exception shall be
> > seen by the Java code. It is not an easy task because all code
> > paths should support it, including JIT and GC.
> >
>
> Agreed. But certainly worth striving for :)
>
> geir
>
>
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 8:51 AM, Gregory Shimansky wrote:
> Geir Magnusson Jr. wrote:
>> The big thing for me is ensuring that we can drive the VM to the
>> limit, and it maintains internal integrity, so applications that
>> are designed to gracefully deal with resource exhaustion can do so
>> w/ confidence that the VM isn't about to crumble out from
>> underneath them.
>
> I agree with Geir that we should try to handle out of C heap
> condition gracefully. The problem is that there is no clearly
> defined contract for many functions that use memory allocation
> about what to do in case of out of memory condition.
>
> To maintain integrity all VM functions which allocate memory from C
> heap should return gracefully all the way up the stack until they
> hit Java code that called them and then OOME exception shall be
> seen by the Java code. It is not an easy task because all code
> paths should support it, including JIT and GC.
>
Agreed. But certainly worth striving for :)
geir
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>> I am not sure that we need to do something about this. The default
>> initial
>> stack size on Windows is 1M,
>
> Yikes! There's our problem on windows...
The same OOME is thrown on Linux as well, but I am not sure if it
happens because of reaching virtual space limit or because of some other
reason. I don't have a good process monitor for Linux to see the number
of running threads and virtual memory size.
>> and that is the recommended init size for real
>> applications. The fact that our threads start with a larger intial stack
>> mapped( default ) than RI is a design issue, it is not a bug. We could
>> start
>> with 2K and create many more threads!
>
> That's right. The fact that the VM crashes and burns is the bug, and a
> serious one, IMO.
>
>> We should not use this test ( which is not a
>> real app ) as guidance to tune the initial stack size. My suggestion
>> is to
>> lower the test duration so that we can create about a 1000( or whatever
>> magic number ) threads at least. That is the stress condition we
>> should test
>> for.
>
> The big thing for me is ensuring that we can drive the VM to the limit,
> and it maintains internal integrity, so applications that are designed
> to gracefully deal with resource exhaustion can do so w/ confidence that
> the VM isn't about to crumble out from underneath them.
I agree with Geir that we should try to handle out of C heap condition
gracefully. The problem is that there is no clearly defined contract for
many functions that use memory allocation about what to do in case of
out of memory condition.
To maintain integrity all VM functions which allocate memory from C heap
should return gracefully all the way up the stack until they hit Java
code that called them and then OOME exception shall be seen by the Java
code. It is not an easy task because all code paths should support it,
including JIT and GC.
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Aleksey Ignatenko <al...@gmail.com>.
On 1/10/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
> On Jan 9, 2007, at 1:14 PM, Rana Dasgupta wrote:
>
> > On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
> >>
> >> On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
> >> >> I've tried to analyze MegaSpawn test on windows and here's what
> >> I found
> >> >> out.
> >> >>
> >> >> OOME is thrown because process virtual size easily gets up to
> >> 2Gb. This
> >> >> happens at about ~1.5k simultaneously running threads. I think it
> >> >> happens because all of virtual process memory is mapped for thread
> >> stacks.
> >> >>
> >>
> >> >Good job! I got the same sort of hunch when I looked at the
> >> source code
> >> did
> >> >not have enough time to pin down specifics. The only guidance I
> >> >found regarding what happens when too many threads are spawned is
> >> the
> >> >following in the java.lang.Thread reference manual, "...specifying a
> >> lower
> >> >[stacksize] value may allow a greater number of threads to exist
> >> >concurrently without throwing an OutOfMemoryError (or other internal
> >> >error)."
> >>
> >> >I think what the above implies is that it is OK for the JVM to
> >> error and
> >> >exit if the app tries to create too many threads. If this is the
> >> case,
> >> it
> >> >sort of looks like we need to clean up the handling of malloc()
> >> errors so
> >> >that the JVM can exit gracefully.
> >
> >
> > I am not sure that we need to do something about this. The default
> > initial
> > stack size on Windows is 1M,
>
> Yikes! There's our problem on windows...
>
> > and that is the recommended init size for real
> > applications. The fact that our threads start with a larger intial
> > stack
> > mapped( default ) than RI is a design issue, it is not a bug. We
> > could start
> > with 2K and create many more threads!
>
> That's right. The fact that the VM crashes and burns is the bug, and
> a serious one, IMO.
>
> > Exactly as Gregory points out,
> > ultimately we will hit virtual memory limits and fail. The reason
> > the RI
> > seems to fail less is that the test ends before running out of virtual
> > memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> > MegaSpawn, with an identical OOME error message and stack dump.
> >
> > We can catch the exception in the test and print a message. But I
> > am not
> > very sure what purpose that would serve. A resource exhaustion
> > exception is
> > a fatal exception and the process is hosed,
>
> No, it's not.
>
> > no real app would be able to do
> > anything more at this point.
>
> That's not true.
>
> > We should not use this test ( which is not a
> > real app ) as guidance to tune the initial stack size. My
> > suggestion is to
> > lower the test duration so that we can create about a 1000( or
> > whatever
> > magic number ) threads at least. That is the stress condition we
> > should test
> > for.
>
> The big thing for me is ensuring that we can drive the VM to the
> limit, and it maintains internal integrity, so applications that are
> designed to gracefully deal with resource exhaustion can do so w/
> confidence that the VM isn't about to crumble out from underneath them.
"VM maintains internal integrity" on OOME situations is a good point. And I
have one very interesting idea how to monitor low memory state in runtime
and change VMs behaviour correspondingly:
the sample code below checks systems memory usage level. So at any moment (
e.g. adding new thread or commiting another portions of Java heap) one can
check that the system memory is almost exhausted. I suppose the number of
such places in drlvm where a lot memory is allocated at once is limited.
void exn_throw_if_exhasted(){
if (port_vmem_usage_rate() > UPPER_MEMORY_BORDER) {
... throw_exception(OOME);
}
}
This functionality will not garantee that we get 100% OOME working, but
variating UPPER_MEMORY_BORDER value we can get low fail rate for such stress
tests like stress.Mix.
Index: vm/port/include/port_vmem.h
===================================================================
--- vm/port/include/port_vmem.h (revision 495134)
+++ vm/port/include/port_vmem.h (working copy)
@@ -94,7 +94,12 @@
*/
APR_DECLARE(size_t *) port_vmem_page_sizes();
+/**
+* Returns % of system memory usage.
+*/
+APR_DECLARE(size_t) port_vmem_usage_rate();
+
#ifdef __cplusplus
}
#endif
Index: vm/port/src/vmem/linux/port_vmem.c
===================================================================
--- vm/port/src/vmem/linux/port_vmem.c (revision 495134)
+++ vm/port/src/vmem/linux/port_vmem.c (working copy)
@@ -20,6 +20,7 @@
*/
#include <sys/mman.h>
+#include <sys/sysinfo.h>
#include <unistd.h>
#include <errno.h>
#include <malloc.h>
@@ -131,6 +132,12 @@
return page_sizes;
}
+APR_DECLARE(size_t) port_vmem_usage_rate() {
+ struct sysinfo info;
+ sysinfo(&info);
+ return (info.totalram - info.freeram)*100/info.totalram;
+}
+
#ifdef __cplusplus
}
#endif
Index: vm/port/src/vmem/win/port_vmem.c
===================================================================
--- vm/port/src/vmem/win/port_vmem.c (revision 495134)
+++ vm/port/src/vmem/win/port_vmem.c (working copy)
@@ -215,6 +215,14 @@
return page_sizes;
}
+APR_DECLARE(size_t) port_vmem_usage_rate(){
+ MEMORYSTATUSEX ms;
+ ms.dwLength = sizeof (ms);
+ GlobalMemoryStatusEx(&ms);
+
+ return ms.dwMemoryLoad;
+}
+
#ifdef __cplusplus
}
#endif
What do you think about the idea?
Best regards,
Aleksey.
> geir
>
>
> > a
> > Thanks,
> > Rana
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 9, 2007, at 1:14 PM, Rana Dasgupta wrote:
> On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
>>
>> On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
>> >> I've tried to analyze MegaSpawn test on windows and here's what
>> I found
>> >> out.
>> >>
>> >> OOME is thrown because process virtual size easily gets up to
>> 2Gb. This
>> >> happens at about ~1.5k simultaneously running threads. I think it
>> >> happens because all of virtual process memory is mapped for thread
>> stacks.
>> >>
>>
>> >Good job! I got the same sort of hunch when I looked at the
>> source code
>> did
>> >not have enough time to pin down specifics. The only guidance I
>> >found regarding what happens when too many threads are spawned is
>> the
>> >following in the java.lang.Thread reference manual, "...specifying a
>> lower
>> >[stacksize] value may allow a greater number of threads to exist
>> >concurrently without throwing an OutOfMemoryError (or other internal
>> >error)."
>>
>> >I think what the above implies is that it is OK for the JVM to
>> error and
>> >exit if the app tries to create too many threads. If this is the
>> case,
>> it
>> >sort of looks like we need to clean up the handling of malloc()
>> errors so
>> >that the JVM can exit gracefully.
>
>
> I am not sure that we need to do something about this. The default
> initial
> stack size on Windows is 1M,
Yikes! There's our problem on windows...
> and that is the recommended init size for real
> applications. The fact that our threads start with a larger intial
> stack
> mapped( default ) than RI is a design issue, it is not a bug. We
> could start
> with 2K and create many more threads!
That's right. The fact that the VM crashes and burns is the bug, and
a serious one, IMO.
> Exactly as Gregory points out,
> ultimately we will hit virtual memory limits and fail. The reason
> the RI
> seems to fail less is that the test ends before running out of virtual
> memory.On my 32 bit RHEL Linux box, RI fails almost every time with
> MegaSpawn, with an identical OOME error message and stack dump.
>
> We can catch the exception in the test and print a message. But I
> am not
> very sure what purpose that would serve. A resource exhaustion
> exception is
> a fatal exception and the process is hosed,
No, it's not.
> no real app would be able to do
> anything more at this point.
That's not true.
> We should not use this test ( which is not a
> real app ) as guidance to tune the initial stack size. My
> suggestion is to
> lower the test duration so that we can create about a 1000( or
> whatever
> magic number ) threads at least. That is the stress condition we
> should test
> for.
The big thing for me is ensuring that we can drive the VM to the
limit, and it maintains internal integrity, so applications that are
designed to gracefully deal with resource exhaustion can do so w/
confidence that the VM isn't about to crumble out from underneath them.
geir
> a
> Thanks,
> Rana
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
On 1/9/07, Weldon Washburn <we...@gmail.com> wrote:
>
> On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
> >> I've tried to analyze MegaSpawn test on windows and here's what I found
> >> out.
> >>
> >> OOME is thrown because process virtual size easily gets up to 2Gb. This
> >> happens at about ~1.5k simultaneously running threads. I think it
> >> happens because all of virtual process memory is mapped for thread
> stacks.
> >>
>
> >Good job! I got the same sort of hunch when I looked at the source code
> did
> >not have enough time to pin down specifics. The only guidance I
> >found regarding what happens when too many threads are spawned is the
> >following in the java.lang.Thread reference manual, "...specifying a
> lower
> >[stacksize] value may allow a greater number of threads to exist
> >concurrently without throwing an OutOfMemoryError (or other internal
> >error)."
>
> >I think what the above implies is that it is OK for the JVM to error and
> >exit if the app tries to create too many threads. If this is the case,
> it
> >sort of looks like we need to clean up the handling of malloc() errors so
> >that the JVM can exit gracefully.
I am not sure that we need to do something about this. The default initial
stack size on Windows is 1M, and that is the recommended init size for real
applications. The fact that our threads start with a larger intial stack
mapped( default ) than RI is a design issue, it is not a bug. We could start
with 2K and create many more threads! Exactly as Gregory points out,
ultimately we will hit virtual memory limits and fail. The reason the RI
seems to fail less is that the test ends before running out of virtual
memory.On my 32 bit RHEL Linux box, RI fails almost every time with
MegaSpawn, with an identical OOME error message and stack dump.
We can catch the exception in the test and print a message. But I am not
very sure what purpose that would serve. A resource exhaustion exception is
a fatal exception and the process is hosed, no real app would be able to do
anything more at this point. We should not use this test ( which is not a
real app ) as guidance to tune the initial stack size. My suggestion is to
lower the test duration so that we can create about a 1000( or whatever
magic number ) threads at least. That is the stress condition we should test
for.
Thanks,
Rana
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 9, 2007, at 10:51 AM, Weldon Washburn wrote:
> On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
>>
>> Geir Magnusson Jr. wrote:
>> > I started a new thread because I think this is really important.
>> >
>> > I've also added a page in the wiki to track this stuff, because
>> I can't
>> > keep it in my head:
>> >
>> > http://wiki.apache.org/harmony/MegaSpawnThreadingBug
>> >
>> > which you can get to from the home page via the "WhiteBoards"
>> section,
>> > intended to be a place where we can work as a team on a
>> whiteboard, with
>> > the intention that once the mini-project is over, we erase...
>
>
> This is a good idea. I still want to put some of the discussion on
> email so
> that we have a permanent record of our investigations. I have some
> thoughts
> inlined below.
>
>>
>> > I think this is a scary scary problem :)
>>
>> I've tried to analyze MegaSpawn test on windows and here's what I
>> found
>> out.
>>
>> OOME is thrown because process virtual size easily gets up to 2Gb.
>> This
>> happens at about ~1.5k simultaneously running threads. I think it
>> happens because all of virtual process memory is mapped for thread
>> stacks.
>>
>> When virtual memory is exhausted all kind of problems may occur.
>> In many
>> places there are assertions that malloc returns non-NULL, and these
>> assertions fail. In some places there are no checks for malloc,
>> and NULL
>> pointer is used for addressing, this also crashes VM.
>
>
> Good job! I got the same sort of hunch when I looked at the source
> code did
> not have enough time to pin down specifics. The only guidance I
> found regarding what happens when too many threads are spawned is the
> following in the java.lang.Thread reference manual, "...specifying
> a lower
> [stacksize] value may allow a greater number of threads to exist
> concurrently without throwing an OutOfMemoryError (or other internal
> error)."
>
> I think what the above implies is that it is OK for the JVM to
> error and
> exit if the app tries to create too many threads. If this is the
> case, it
> sort of looks like we need to clean up the handling of malloc()
> errors so
> that the JVM can exit gracefully.
Well - I think that we should strive to maintain an internally
consistent VM, throw an OOM, and let the app decide. There are
situations where with a solid VM, you can deal w/ the OOM at the app
level.
>
> Another approach would be to throw something like a,
> "TooManyThreadsAtOnceException" and keep running the app. I can't
> find
> anything like this kind of exception. Its probably not an option.
No :)
>
> Another approach would be to make Thread.start() method wait until
> there are
> enough resources to create a new thread. Most likely the app would
> hang
> mysteriously without warning. This is probably not an option either.
Nope :)
>
> Another item we need to discuss is what are the Q1/Q2 goals for max
> number
> of threads supported? It seems we can do lots of useful stuff with
> a max of
> 1500 threads. The useful stuff being items like the bringup of
> enterprise
> apps, fixing stability problems...
I don't mind a suboptimal # of concurrent threads - we can work on
that over time. The fact that the VM falls over dead scares the
bejeezus out of me.
geir
>
>
> I tried to watch Sun implementation and it looks like they map smaller
>> amounts of memory for thread stacks. Maybe they map only initial
>> stack
>> memory somehow and allow it to grow later (although I don't quite
>> understand how it is possible in continuous address space). When
>> Sun VM
>> executes this test it created up to ~6k simultaneously running
>> threads
>> and process size at the same moment was smaller than 2Gb.
>>
>> I think the same problem may happen on Linux because it spills out
>> OOMEs
>> on Ubuntu as well.
>>
>> If somehow test doesn't crash on failed mallocs and gets to the
>> shutdown
>> stage and hangs with 2 or more dead locked threads. So far I didn't
>> quite understand how they lock each other.
>>
>> --
>> Gregory
>>
>>
>
>
> --
> Weldon Washburn
> Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Weldon Washburn <we...@gmail.com>.
On 1/9/07, Gregory Shimansky <gs...@gmail.com> wrote:
>
> Geir Magnusson Jr. wrote:
> > I started a new thread because I think this is really important.
> >
> > I've also added a page in the wiki to track this stuff, because I can't
> > keep it in my head:
> >
> > http://wiki.apache.org/harmony/MegaSpawnThreadingBug
> >
> > which you can get to from the home page via the "WhiteBoards" section,
> > intended to be a place where we can work as a team on a whiteboard, with
> > the intention that once the mini-project is over, we erase...
This is a good idea. I still want to put some of the discussion on email so
that we have a permanent record of our investigations. I have some thoughts
inlined below.
>
> > I think this is a scary scary problem :)
>
> I've tried to analyze MegaSpawn test on windows and here's what I found
> out.
>
> OOME is thrown because process virtual size easily gets up to 2Gb. This
> happens at about ~1.5k simultaneously running threads. I think it
> happens because all of virtual process memory is mapped for thread stacks.
>
> When virtual memory is exhausted all kind of problems may occur. In many
> places there are assertions that malloc returns non-NULL, and these
> assertions fail. In some places there are no checks for malloc, and NULL
> pointer is used for addressing, this also crashes VM.
Good job! I got the same sort of hunch when I looked at the source code did
not have enough time to pin down specifics. The only guidance I
found regarding what happens when too many threads are spawned is the
following in the java.lang.Thread reference manual, "...specifying a lower
[stacksize] value may allow a greater number of threads to exist
concurrently without throwing an OutOfMemoryError (or other internal
error)."
I think what the above implies is that it is OK for the JVM to error and
exit if the app tries to create too many threads. If this is the case, it
sort of looks like we need to clean up the handling of malloc() errors so
that the JVM can exit gracefully.
Another approach would be to throw something like a,
"TooManyThreadsAtOnceException" and keep running the app. I can't find
anything like this kind of exception. Its probably not an option.
Another approach would be to make Thread.start() method wait until there are
enough resources to create a new thread. Most likely the app would hang
mysteriously without warning. This is probably not an option either.
Another item we need to discuss is what are the Q1/Q2 goals for max number
of threads supported? It seems we can do lots of useful stuff with a max of
1500 threads. The useful stuff being items like the bringup of enterprise
apps, fixing stability problems...
I tried to watch Sun implementation and it looks like they map smaller
> amounts of memory for thread stacks. Maybe they map only initial stack
> memory somehow and allow it to grow later (although I don't quite
> understand how it is possible in continuous address space). When Sun VM
> executes this test it created up to ~6k simultaneously running threads
> and process size at the same moment was smaller than 2Gb.
>
> I think the same problem may happen on Linux because it spills out OOMEs
> on Ubuntu as well.
>
> If somehow test doesn't crash on failed mallocs and gets to the shutdown
> stage and hangs with 2 or more dead locked threads. So far I didn't
> quite understand how they lock each other.
>
> --
> Gregory
>
>
--
Weldon Washburn
Intel Enterprise Solutions Software Division
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Salikh Zakirov <Sa...@Intel.com>.
Gregory Shimansky wrote:
> Today investigation of the hanging threads at shutdown have 2 different
> reasons. 1st one was found by Salikh and he wrote his comments in
> HARMONY-2963. The bug happened because the counter of non-daemon threads
> increased before a thread was created. If a thread failed to be created
> because of no memory, this counter was not updated.
>
> Another reason for hanging threads is that they wait in Thread.start().
> When a new thread is started, it has to notify a lock object, in order
> to signal the parent thread that it has been created. This notification
> is sent from java code of the Thread before user code is executed.
>
> But thread manager has some native code too which is ran before java
> code of the newly started thread. This native code tried to set up some
> thread state like new JNI environment and other stuff, and this requires
> allocation of new memory. If allocation of new memory fails, this native
> code of the newly created thread tries to return an error which is not
> seen anywhere (since this is the code which is the first function of the
> new thread), so it is not noticed. But since native code of the new
> thread finishes silently, it never runs the Java code which should do
> monitor notification, so monitor is not notified. So the parent thread
> just waits infinitely.
For the record, the following fix changes VM behaviour from silently
ignoring errors during late thread startup to assertion.
(so the MegaSpawn test crashes instead of hanging due to wait() which will
never be notified)
--- vm/thread/src/thread_java_basic.c
+++ vm/thread/src/thread_java_basic.c
@@ -81,7 +81,7 @@ int wrapper_proc(void *arg) {
java_thread = jvmti_thread->thread_object;
status = vm_attach(data->java_vm, &jni_env);
- if (status != JNI_OK) return TM_ERROR_INTERNAL;
+ assert(status == JNI_OK); // XXX
jvmti_thread->jenv = jni_env;
jvmti_thread->daemon = data->daemon;
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 12, 2007, at 5:02 AM, Salikh Zakirov wrote:
>
>> 2007/1/11, Gregory Shimansky <gs...@gmail.com>:
>>> Another reason for hanging threads is that they wait in
>>> Thread.start().
>>> When a new thread is started, it has to notify a lock object, in
>>> order
>>> to signal the parent thread that it has been created. This
>>> notification
>>> is sent from java code of the Thread before user code is executed.
>
> Alexey Varlamov replied:
>> IMO the problem is in this design rather than its impl. The
>> specification for j.l.Thread.start() does not require that execution
>> of run() begins strictly before the start() returned.
>
> This requirement is implicated by specification of Thread.isAlive():
>
> Tests if this thread is alive. A thread is alive
> if it has been started and has not yet died.
>
>
I wonder how long it took the EG to come up with that...
geir
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Yes.
Nikolay/Artem/Andrey : Can you comment on this?
:)
geir
On Jan 12, 2007, at 8:12 PM, Rana Dasgupta wrote:
> Would it be possible to request Nikolay/Artem/Andrey to comment on
> this,
> since the reason does not seem obvious ?
>
> On 1/12/07, Salikh Zakirov <Sa...@intel.com> wrote:
>>
>> Gregory Shimansky wrote:
>>
>> >>> This requirement is implicated by specification of
>> Thread.isAlive():
>> >>>
>> >>> Tests if this thread is alive. A thread is alive
>> >>> if it has been started and has not yet died.
>> >>>
>> >> I see no problem here: isAlive() must return true after the
>> start()
>> >> returned, so the thread state is consistent, and that's all.
>> >
>> > I've made an experiment with the patch below. I've removed wait/
>> notify
>> > mechanism and set isAlive in the start method right after
>> > VMThreadManager.start is called. So isAlive is set twice, on the
>> stack
>> > of parent thread, and on the stack of created thread before calling
>> > java. This seem to work just fine, all tests have passed,
>> eclipse works
>> > too.
>> >
>> > I wonder why all this wait/notify stuff was written at all. I
>> think we
>> > can get rid of it.
>>
>>
>> I've just noticed, that very similar code was there before committing
>> HARMONY-1648. Unfortunately, Nikolay did not describe implications of
>> synchronous thread start on stop() method.
>>
>>
>> --
>> Salikh
>>
>>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
Would it be possible to request Nikolay/Artem/Andrey to comment on this,
since the reason does not seem obvious ?
On 1/12/07, Salikh Zakirov <Sa...@intel.com> wrote:
>
> Gregory Shimansky wrote:
>
> >>> This requirement is implicated by specification of Thread.isAlive():
> >>>
> >>> Tests if this thread is alive. A thread is alive
> >>> if it has been started and has not yet died.
> >>>
> >> I see no problem here: isAlive() must return true after the start()
> >> returned, so the thread state is consistent, and that's all.
> >
> > I've made an experiment with the patch below. I've removed wait/notify
> > mechanism and set isAlive in the start method right after
> > VMThreadManager.start is called. So isAlive is set twice, on the stack
> > of parent thread, and on the stack of created thread before calling
> > java. This seem to work just fine, all tests have passed, eclipse works
> > too.
> >
> > I wonder why all this wait/notify stuff was written at all. I think we
> > can get rid of it.
>
>
> I've just noticed, that very similar code was there before committing
> HARMONY-1648. Unfortunately, Nikolay did not describe implications of
> synchronous thread start on stop() method.
>
>
> --
> Salikh
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Salikh Zakirov <Sa...@Intel.com>.
Gregory Shimansky wrote:
>>> This requirement is implicated by specification of Thread.isAlive():
>>>
>>> Tests if this thread is alive. A thread is alive
>>> if it has been started and has not yet died.
>>>
>> I see no problem here: isAlive() must return true after the start()
>> returned, so the thread state is consistent, and that's all.
>
> I've made an experiment with the patch below. I've removed wait/notify
> mechanism and set isAlive in the start method right after
> VMThreadManager.start is called. So isAlive is set twice, on the stack
> of parent thread, and on the stack of created thread before calling
> java. This seem to work just fine, all tests have passed, eclipse works
> too.
>
> I wonder why all this wait/notify stuff was written at all. I think we
> can get rid of it.
I've just noticed, that very similar code was there before committing
HARMONY-1648. Unfortunately, Nikolay did not describe implications of
synchronous thread start on stop() method.
--
Salikh
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Rana Dasgupta <rd...@gmail.com>.
There is some history in this that I read through in Harmony issues 1789,
2219, and now 2963. There may be more.
Possibly the interpretation that the child thread is in run state for
isAlive() to be true is an overinterpretation. The spec seems to intend this
to be implemantation dependent, I could not find any specific reference to
this. The wait/notify/interrupt status forwarding in the implementation seem
to be all based on this assumption.
On 1/12/07, Gregory Shimansky <gs...@gmail.com> wrote:
>
> Alexey Varlamov wrote:
> > 2007/1/12, Salikh Zakirov <Sa...@intel.com>:
> >>
> >> > 2007/1/11, Gregory Shimansky <gs...@gmail.com>:
> >> >> Another reason for hanging threads is that they wait in
> >> Thread.start().
> >> >> When a new thread is started, it has to notify a lock object, in
> order
> >> >> to signal the parent thread that it has been created. This
> >> notification
> >> >> is sent from java code of the Thread before user code is executed.
> >>
> >> Alexey Varlamov replied:
> >> > IMO the problem is in this design rather than its impl. The
> >> > specification for j.l.Thread.start() does not require that execution
> >> > of run() begins strictly before the start() returned.
> >>
> >> This requirement is implicated by specification of Thread.isAlive():
> >>
> >> Tests if this thread is alive. A thread is alive
> >> if it has been started and has not yet died.
> >>
> > I see no problem here: isAlive() must return true after the start()
> > returned, so the thread state is consistent, and that's all.
>
> I've made an experiment with the patch below. I've removed wait/notify
> mechanism and set isAlive in the start method right after
> VMThreadManager.start is called. So isAlive is set twice, on the stack
> of parent thread, and on the stack of created thread before calling
> java. This seem to work just fine, all tests have passed, eclipse works
> too.
>
> I wonder why all this wait/notify stuff was written at all. I think we
> can get rid of it.
>
>
> Index: vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java
> ===================================================================
> --- vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java
> (revision 495565)
> +++ vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java (working
> copy)
> @@ -663,7 +663,7 @@
> synchronized (lock) {
> this.isAlive = true;
> this.started = true;
> - lock.notifyAll();
> +// lock.notifyAll();
> }
>
> run();
> @@ -739,19 +739,21 @@
> if (VMThreadManager.start(this, stackSize, daemon,
> priority) != 0)
> {
> throw new OutOfMemoryError("Failed to create new
> thread");
> }
> -
> - boolean interrupted = false;
> - while(!this.started) {
> - try {
> - lock.wait();
> - } catch (InterruptedException e) {
> - interrupted = true;
> - }
> - }
>
> - if (interrupted) {
> - Thread.currentThread().interrupt();
> - }
> + this.started = true;
> + this.isAlive = true;
> +// boolean interrupted = false;
> +// while(!this.started) {
> +// try {
> +// lock.wait();
> +// } catch (InterruptedException e) {
> +// interrupted = true;
> +// }
> +// }
> +
> +// if (interrupted) {
> +// Thread.currentThread().interrupt();
> +// }
> }
> }
>
> --
> Gregory
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Alexey Varlamov wrote:
> 2007/1/12, Salikh Zakirov <Sa...@intel.com>:
>>
>> > 2007/1/11, Gregory Shimansky <gs...@gmail.com>:
>> >> Another reason for hanging threads is that they wait in
>> Thread.start().
>> >> When a new thread is started, it has to notify a lock object, in order
>> >> to signal the parent thread that it has been created. This
>> notification
>> >> is sent from java code of the Thread before user code is executed.
>>
>> Alexey Varlamov replied:
>> > IMO the problem is in this design rather than its impl. The
>> > specification for j.l.Thread.start() does not require that execution
>> > of run() begins strictly before the start() returned.
>>
>> This requirement is implicated by specification of Thread.isAlive():
>>
>> Tests if this thread is alive. A thread is alive
>> if it has been started and has not yet died.
>>
> I see no problem here: isAlive() must return true after the start()
> returned, so the thread state is consistent, and that's all.
I've made an experiment with the patch below. I've removed wait/notify
mechanism and set isAlive in the start method right after
VMThreadManager.start is called. So isAlive is set twice, on the stack
of parent thread, and on the stack of created thread before calling
java. This seem to work just fine, all tests have passed, eclipse works too.
I wonder why all this wait/notify stuff was written at all. I think we
can get rid of it.
Index: vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java
===================================================================
--- vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java
(revision 495565)
+++ vm/vmcore/src/kernel_classes/javasrc/java/lang/Thread.java (working
copy)
@@ -663,7 +663,7 @@
synchronized (lock) {
this.isAlive = true;
this.started = true;
- lock.notifyAll();
+// lock.notifyAll();
}
run();
@@ -739,19 +739,21 @@
if (VMThreadManager.start(this, stackSize, daemon,
priority) != 0)
{
throw new OutOfMemoryError("Failed to create new thread");
}
-
- boolean interrupted = false;
- while(!this.started) {
- try {
- lock.wait();
- } catch (InterruptedException e) {
- interrupted = true;
- }
- }
- if (interrupted) {
- Thread.currentThread().interrupt();
- }
+ this.started = true;
+ this.isAlive = true;
+// boolean interrupted = false;
+// while(!this.started) {
+// try {
+// lock.wait();
+// } catch (InterruptedException e) {
+// interrupted = true;
+// }
+// }
+
+// if (interrupted) {
+// Thread.currentThread().interrupt();
+// }
}
}
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Alexey Varlamov <al...@gmail.com>.
2007/1/12, Salikh Zakirov <Sa...@intel.com>:
>
> > 2007/1/11, Gregory Shimansky <gs...@gmail.com>:
> >> Another reason for hanging threads is that they wait in Thread.start().
> >> When a new thread is started, it has to notify a lock object, in order
> >> to signal the parent thread that it has been created. This notification
> >> is sent from java code of the Thread before user code is executed.
>
> Alexey Varlamov replied:
> > IMO the problem is in this design rather than its impl. The
> > specification for j.l.Thread.start() does not require that execution
> > of run() begins strictly before the start() returned.
>
> This requirement is implicated by specification of Thread.isAlive():
>
> Tests if this thread is alive. A thread is alive
> if it has been started and has not yet died.
>
I see no problem here: isAlive() must return true after the start()
returned, so the thread state is consistent, and that's all.
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Salikh Zakirov <Sa...@Intel.com>.
> 2007/1/11, Gregory Shimansky <gs...@gmail.com>:
>> Another reason for hanging threads is that they wait in Thread.start().
>> When a new thread is started, it has to notify a lock object, in order
>> to signal the parent thread that it has been created. This notification
>> is sent from java code of the Thread before user code is executed.
Alexey Varlamov replied:
> IMO the problem is in this design rather than its impl. The
> specification for j.l.Thread.start() does not require that execution
> of run() begins strictly before the start() returned.
This requirement is implicated by specification of Thread.isAlive():
Tests if this thread is alive. A thread is alive
if it has been started and has not yet died.
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Alexey Varlamov <al...@gmail.com>.
2007/1/11, Gregory Shimansky <gs...@gmail.com>:
> Geir Magnusson Jr. wrote:
> >
> > On Jan 10, 2007, at 9:00 AM, Gregory Shimansky wrote:
> >
> >> Geir Magnusson Jr. wrote:
> >>>> I think the same problem may happen on Linux because it spills out
> >>>> OOMEs on Ubuntu as well.
> >>>>
> >>>> If somehow test doesn't crash on failed mallocs and gets to the
> >>>> shutdown stage and hangs with 2 or more dead locked threads. So far
> >>>> I didn't quite understand how they lock each other.
> >>> Cool - thanks. If you have a free second, could you note this on the
> >>> wiki page so we don't forget?
> >>
> >> I think it is better to track this with JIRA. AFAIU is not a stress
> >> conditions issue and so it is a normal bug which should be found and
> >> fixed. I created a new JIRA HARMONY-2963 which is subtask for
> >> HARMONY-2803 where Weldon attached his MegaSpawn test.
> >
> > Agreed that a JIRA is important - I just wanted to make sure that we
> > added it somehow to the whiteboard so we had a complete picture of
> > things related to this problem.
>
> Today investigation of the hanging threads at shutdown have 2 different
> reasons. 1st one was found by Salikh and he wrote his comments in
> HARMONY-2963. The bug happened because the counter of non-daemon threads
> increased before a thread was created. If a thread failed to be created
> because of no memory, this counter was not updated.
Good catch!
> Another reason for hanging threads is that they wait in Thread.start().
> When a new thread is started, it has to notify a lock object, in order
> to signal the parent thread that it has been created. This notification
> is sent from java code of the Thread before user code is executed.
IMO the problem is in this design rather than its impl. The
specification for j.l.Thread.start() does not require that execution
of run() begins strictly before the start() returned. So in my
perception that loop with waiting is unnecessary complication, it is
enough to launch new native thread and let OS schedule.
> But thread manager has some native code too which is ran before java
> code of the newly started thread. This native code tried to set up some
> thread state like new JNI environment and other stuff, and this requires
> allocation of new memory. If allocation of new memory fails, this native
> code of the newly created thread tries to return an error which is not
> seen anywhere (since this is the code which is the first function of the
> new thread), so it is not noticed. But since native code of the new
> thread finishes silently, it never runs the Java code which should do
> monitor notification, so monitor is not notified. So the parent thread
> just waits infinitely.
I suppose a new thread may fail to run Java code for a number of
reasons, e.g. failed JIT compilation, malfunctioning TI agent, etc -
not only due to memory exhaustion. We cannot guarantee hang-free
behavior, this is design-inherent issue.
> To fix this bug I think it is necessary to get rid of error conditions
> in the newly created threads. I think it is necessary to allocate all
> necessary state before a new thread is started, so if these resources
> cannot be allocated, an error should be returned to the parent thread,
> and it won't wait infinitely on new thread start notification.
Well, this would not resolve all possible reasons for the infinite
wait, as noted above. Besides, this creates another kind of memory
waste/leak if a thread is never run. My proposal is opposite, do all
native allocations no earlier than start() is invoked, naturally throw
OOME if any, and rely on OS scheduler to run the new thread
concurrently.
BTW, AFAIU in the current design TM never frees native memory
allocated for new thread data, though sometimes reuses it for
collected threads. There are related HARMONY-2437 and HARMONY-2742. I
don't think this is the reason of MegaSpawn crashes, but yet another
issue of the kind.
>
> --
> Gregory
>
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>
> On Jan 10, 2007, at 9:00 AM, Gregory Shimansky wrote:
>
>> Geir Magnusson Jr. wrote:
>>>> I think the same problem may happen on Linux because it spills out
>>>> OOMEs on Ubuntu as well.
>>>>
>>>> If somehow test doesn't crash on failed mallocs and gets to the
>>>> shutdown stage and hangs with 2 or more dead locked threads. So far
>>>> I didn't quite understand how they lock each other.
>>> Cool - thanks. If you have a free second, could you note this on the
>>> wiki page so we don't forget?
>>
>> I think it is better to track this with JIRA. AFAIU is not a stress
>> conditions issue and so it is a normal bug which should be found and
>> fixed. I created a new JIRA HARMONY-2963 which is subtask for
>> HARMONY-2803 where Weldon attached his MegaSpawn test.
>
> Agreed that a JIRA is important - I just wanted to make sure that we
> added it somehow to the whiteboard so we had a complete picture of
> things related to this problem.
Today investigation of the hanging threads at shutdown have 2 different
reasons. 1st one was found by Salikh and he wrote his comments in
HARMONY-2963. The bug happened because the counter of non-daemon threads
increased before a thread was created. If a thread failed to be created
because of no memory, this counter was not updated.
Another reason for hanging threads is that they wait in Thread.start().
When a new thread is started, it has to notify a lock object, in order
to signal the parent thread that it has been created. This notification
is sent from java code of the Thread before user code is executed.
But thread manager has some native code too which is ran before java
code of the newly started thread. This native code tried to set up some
thread state like new JNI environment and other stuff, and this requires
allocation of new memory. If allocation of new memory fails, this native
code of the newly created thread tries to return an error which is not
seen anywhere (since this is the code which is the first function of the
new thread), so it is not noticed. But since native code of the new
thread finishes silently, it never runs the Java code which should do
monitor notification, so monitor is not notified. So the parent thread
just waits infinitely.
To fix this bug I think it is necessary to get rid of error conditions
in the newly created threads. I think it is necessary to allocate all
necessary state before a new thread is started, so if these resources
cannot be allocated, an error should be returned to the parent thread,
and it won't wait infinitely on new thread start notification.
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 10, 2007, at 9:00 AM, Gregory Shimansky wrote:
> Geir Magnusson Jr. wrote:
>>> I think the same problem may happen on Linux because it spills
>>> out OOMEs on Ubuntu as well.
>>>
>>> If somehow test doesn't crash on failed mallocs and gets to the
>>> shutdown stage and hangs with 2 or more dead locked threads. So
>>> far I didn't quite understand how they lock each other.
>> Cool - thanks. If you have a free second, could you note this on
>> the wiki page so we don't forget?
>
> I think it is better to track this with JIRA. AFAIU is not a stress
> conditions issue and so it is a normal bug which should be found
> and fixed. I created a new JIRA HARMONY-2963 which is subtask for
> HARMONY-2803 where Weldon attached his MegaSpawn test.
Agreed that a JIRA is important - I just wanted to make sure that we
added it somehow to the whiteboard so we had a complete picture of
things related to this problem.
geir
>
> --
> Gregory
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>> I think the same problem may happen on Linux because it spills out
>> OOMEs on Ubuntu as well.
>>
>> If somehow test doesn't crash on failed mallocs and gets to the
>> shutdown stage and hangs with 2 or more dead locked threads. So far I
>> didn't quite understand how they lock each other.
>
> Cool - thanks. If you have a free second, could you note this on the
> wiki page so we don't forget?
I think it is better to track this with JIRA. AFAIU is not a stress
conditions issue and so it is a normal bug which should be found and
fixed. I created a new JIRA HARMONY-2963 which is subtask for
HARMONY-2803 where Weldon attached his MegaSpawn test.
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Jan 9, 2007, at 9:02 AM, Gregory Shimansky wrote:
> Geir Magnusson Jr. wrote:
>> I started a new thread because I think this is really important.
>> I've also added a page in the wiki to track this stuff, because I
>> can't keep it in my head:
>> http://wiki.apache.org/harmony/MegaSpawnThreadingBug
>> which you can get to from the home page via the "WhiteBoards"
>> section, intended to be a place where we can work as a team on a
>> whiteboard, with the intention that once the mini-project is over,
>> we erase...
>> I think this is a scary scary problem :)
>
> I've tried to analyze MegaSpawn test on windows and here's what I
> found out.
>
> OOME is thrown because process virtual size easily gets up to 2Gb.
> This happens at about ~1.5k simultaneously running threads. I think
> it happens because all of virtual process memory is mapped for
> thread stacks.
>
> When virtual memory is exhausted all kind of problems may occur. In
> many places there are assertions that malloc returns non-NULL, and
> these assertions fail. In some places there are no checks for
> malloc, and NULL pointer is used for addressing, this also crashes VM.
>
This is actually good news (I thinK), as I'd rather be running out of
heap rather than trashing it.
This is also useful for hardening - we should spend some time finding
places where we aren't checking mallocs and such..
> I tried to watch Sun implementation and it looks like they map
> smaller amounts of memory for thread stacks. Maybe they map only
> initial stack memory somehow and allow it to grow later (although I
> don't quite understand how it is possible in continuous address
> space). When Sun VM executes this test it created up to ~6k
> simultaneously running threads and process size at the same moment
> was smaller than 2Gb.
>
> I think the same problem may happen on Linux because it spills out
> OOMEs on Ubuntu as well.
>
> If somehow test doesn't crash on failed mallocs and gets to the
> shutdown stage and hangs with 2 or more dead locked threads. So far
> I didn't quite understand how they lock each other.
Cool - thanks. If you have a free second, could you note this on the
wiki page so we don't forget?
geir
>
> --
> Gregory
>
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
> I started a new thread because I think this is really important.
>
> I've also added a page in the wiki to track this stuff, because I can't
> keep it in my head:
>
> http://wiki.apache.org/harmony/MegaSpawnThreadingBug
>
> which you can get to from the home page via the "WhiteBoards" section,
> intended to be a place where we can work as a team on a whiteboard, with
> the intention that once the mini-project is over, we erase...
>
> I think this is a scary scary problem :)
I've tried to analyze MegaSpawn test on windows and here's what I found out.
OOME is thrown because process virtual size easily gets up to 2Gb. This
happens at about ~1.5k simultaneously running threads. I think it
happens because all of virtual process memory is mapped for thread stacks.
When virtual memory is exhausted all kind of problems may occur. In many
places there are assertions that malloc returns non-NULL, and these
assertions fail. In some places there are no checks for malloc, and NULL
pointer is used for addressing, this also crashes VM.
I tried to watch Sun implementation and it looks like they map smaller
amounts of memory for thread stacks. Maybe they map only initial stack
memory somehow and allow it to grow later (although I don't quite
understand how it is possible in continuous address space). When Sun VM
executes this test it created up to ~6k simultaneously running threads
and process size at the same moment was smaller than 2Gb.
I think the same problem may happen on Linux because it spills out OOMEs
on Ubuntu as well.
If somehow test doesn't crash on failed mallocs and gets to the shutdown
stage and hangs with 2 or more dead locked threads. So far I didn't
quite understand how they lock each other.
--
Gregory
Re: [drlvm] stress.Mix / MegaSpawn threading bug
Posted by Elena Semukhina <el...@gmail.com>.
As it appears from other threads, stress.Mix behaves differently when it
runs from command line or via build test. When I reported that it always
passed for me on em64t, I meant running from command line. I've just tried
to run it via build test and have seen intermittent failures. I'll add the
test logs to HARMONY-2772 and update the wiki page.
Elena
On 1/9/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
> I started a new thread because I think this is really important.
>
> I've also added a page in the wiki to track this stuff, because I
> can't keep it in my head:
>
> http://wiki.apache.org/harmony/MegaSpawnThreadingBug
>
> which you can get to from the home page via the "WhiteBoards"
> section, intended to be a place where we can work as a team on a
> whiteboard, with the intention that once the mini-project is over, we
> erase...
>
> I think this is a scary scary problem :)
>
> geir
>
>
--
Thanks,
Elena