[drlvm][threading] H3289 -- some JVMTI questions

[Weldon Washburn <we...@gmail.com>]

All,
I thought it would be good to discuss H3289 on dev list instead of buried
inside of the JIRA.  Gregory mentioned in the JIRA that JVMTI needs to be
able to grab the "Lock" instance in Thread.java, do some work, then release
the instance of "Lock".  I think this would require native JVMTI code to
upcall to java code that would do something like:

//java code
void synchronized upcall (int command)  {
    downcall_into_native_C_code_to_do some_JVMTI(command);
}

The above is rather convoluted.  Maybe a simpler approach would be to use
java.util.concurrent.locks.  The code would look something like:

//native code

upcall java code ---->>    java.util.concurrent.locks.Lock();

----  normal JVMTI C code ----

upcall java code ---->>    java.util.concurrent.locks.Unlock();

Comments on the above?

On another topic, do we really have to use Thread.stop() in the
implementation of vm_shutdown_stop_java_threads() ?  This seems way
too hard.   All we really want to do is give back all the OS resources.  We
could simply call OS kill on each specific thread, return all memory via
"free()", return all file handles, etc.  Note if there was no need to allow
multiple JVMs in a single address space to come and go, we could simplify
the above and basically just call process exit().
-- 
Weldon Washburn
Intel Enterprise Solutions Software Division

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Gregory Shimansky <gs...@gmail.com>]

Weldon Washburn wrote:
> All,
> I thought it would be good to discuss H3289 on dev list instead of buried
> inside of the JIRA.  Gregory mentioned in the JIRA that JVMTI needs to be
> able to grab the "Lock" instance in Thread.java, do some work, then release
> the instance of "Lock".  I think this would require native JVMTI code to
> upcall to java code that would do something like:
> 
> //java code
> void synchronized upcall (int command)  {
>    downcall_into_native_C_code_to_do some_JVMTI(command);
> }
> 
> The above is rather convoluted.  Maybe a simpler approach would be to use
> java.util.concurrent.locks.  The code would look something like:
> 
> //native code
> 
> upcall java code ---->>    java.util.concurrent.locks.Lock();
> 
> ----  normal JVMTI C code ----
> 
> upcall java code ---->>    java.util.concurrent.locks.Unlock();
> 
> Comments on the above?

JVMTI is the API which is called by a user agent and in most cases 
specification allows the agent to call any kind of functions in any 
conditions in the live phase of VM. The calls to agent are done on 
events [1] which this agent is subscribed to. We cannot guard all of 
such events by locking a thread lock, such code may very easily lead to 
deadlocks too.

[1] http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html#EventIndex

> On another topic, do we really have to use Thread.stop() in the
> implementation of vm_shutdown_stop_java_threads() ?  This seems way
> too hard.   All we really want to do is give back all the OS resources.  We
> could simply call OS kill on each specific thread, return all memory via
> "free()", return all file handles, etc.  Note if there was no need to allow
> multiple JVMs in a single address space to come and go, we could simplify
> the above and basically just call process exit().

-- 
Gregory

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Gregory Shimansky <gs...@gmail.com>]

Weldon Washburn wrote:
> On 3/16/07, Salikh Zakirov <Sa...@intel.com> wrote:
>>
>> Weldon Washburn wrote:
>> > On another topic, do we really have to use Thread.stop() in the
>> > implementation of vm_shutdown_stop_java_threads() ?  This seems way
>> > too hard.   All we really want to do is give back all the OS
>> resources.  We
>> > could simply call OS kill on each specific thread, return all memory 
>> via
>> > "free()", return all file handles, etc.  Note if there was no need to
>> allow
>> > multiple JVMs in a single address space to come and go, we could
>> simplify
>> > the above and basically just call process exit().
>>
>> HARMONY-3381 (already committed) replaces Thread.stop-like shutdown
>> algorithm
>> to the following one
>>
>> 1) set a shutdown callback, which executes hythread_exit() (which
>> delegates to
>> either pthread_exit() or ExitThread())
>>
>> 2) wait for a little while -- this is a little bit "unscientific" and
>> assumes,
>> that all actively running threads will reach a callback safepoint within
>> that
>> "little while". If assumption holds true, the remaining threads are 
>> highly
>> likely to be unactive threads, i.e. sleeping, waiting or blocked.
>> The key about a little while is that if all threads has terminated
>> promptly, we
>> don't need to wait any longer at all. It is currently implemented using
>> hythread_join() with timeout.
>>
>> 3) on the third step the remaining threads are terminated using
>> hythread_cancel() (which delegates to either of pthread_cancel() or
>> TerminateThread()) -- This step is inherently dangerous on Windows, as
>> terminated thread may be holding system-wide malloc lock, and subsequent
>> free()
>> on a main thread will deadlock.  Fortunately, the unscientific assumption
>> of the
>> step (2) makes this rather unprobable.
> 
> 
> 
> Yes, agreed this is dangerous and rather unprobable.  I reluctantly agree
> with this approach as we have other, more pressing bugs to fix.  
> Besides, it
> is fairly easy to know if we hit problems in this area since the bug will
> occur only during shutdown.

I think that in the future we can make ourself free from MSVC runtime 
which implements malloc with global lock, and then this problem will 
disappear. This will require implementing a VM-wide memory manager which 
isn't an easy task especially since we have JIT and GC as separate 
components.

> Using pthread_cancel() on linux has its own set of advantages and
>> disadvantages. By default, threads are created with cancellability 
>> enabled
>> and
>> in "deferred cancellability" state. It means that pthread_cancel() will
>> only
>> terminate thread when it reaches an explicit cancellation point, such as
>> pthread_cond_wait(). In this way, pthread_cancel() does not suffer from
>> the
>> malloc-deadlock issue.
>>
>> However, deferred cancellability brings its own problems, for example, as
>> pthread_mutex_lock is not a cancellation point, a pair of deadlocked
>> daemon
>> threads will still be alive during VM shutdown, and could probably result
>> in
>> EINVAL return and subsequent assertion failure if the monitors are
>> destroyed in
>> the main shutdown thread.
>>
>> So, the shutdown algorithm has been improved compared to the former
>> exception-based approach, like Thread.stop(), but still is not perfect.
>> Eugene and me discussed this and agreed to keep an open eye on shutdown
>> bugs.
> 
> 
> 
> The shutdown code has definitely been improved.  If it works for the 
> apps we
> care about, then its good enough for the time being.
> 
> There is still a field for experiments and possible improvements, e.g. 
> using
>> asynchronous cancellability or even using pthread_kill instead of
>> pthread_cancel to implement immediate thread termination.
> 
> 
> 
> Hmm... resource reclaimation is always the worst part of building systems
> software.  It will probably take a few iterations to get it right.  Somehow
> Linux can core dump a bad process and successfully recover all dangling 
> file
> handles, network sockets, page directories, thread info blocks, etc.  Maybe
> we look at how OS's do resource recovery for some ideas.
> 
> -- 
>> Salikh Zakirov
>>
>>
> 
> 


-- 
Gregory

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Weldon Washburn <we...@gmail.com>]

On 3/16/07, Salikh Zakirov <Sa...@intel.com> wrote:
>
> Weldon Washburn wrote:
> > On another topic, do we really have to use Thread.stop() in the
> > implementation of vm_shutdown_stop_java_threads() ?  This seems way
> > too hard.   All we really want to do is give back all the OS
> resources.  We
> > could simply call OS kill on each specific thread, return all memory via
> > "free()", return all file handles, etc.  Note if there was no need to
> allow
> > multiple JVMs in a single address space to come and go, we could
> simplify
> > the above and basically just call process exit().
>
> HARMONY-3381 (already committed) replaces Thread.stop-like shutdown
> algorithm
> to the following one
>
> 1) set a shutdown callback, which executes hythread_exit() (which
> delegates to
> either pthread_exit() or ExitThread())
>
> 2) wait for a little while -- this is a little bit "unscientific" and
> assumes,
> that all actively running threads will reach a callback safepoint within
> that
> "little while". If assumption holds true, the remaining threads are highly
> likely to be unactive threads, i.e. sleeping, waiting or blocked.
> The key about a little while is that if all threads has terminated
> promptly, we
> don't need to wait any longer at all. It is currently implemented using
> hythread_join() with timeout.
>
> 3) on the third step the remaining threads are terminated using
> hythread_cancel() (which delegates to either of pthread_cancel() or
> TerminateThread()) -- This step is inherently dangerous on Windows, as
> terminated thread may be holding system-wide malloc lock, and subsequent
> free()
> on a main thread will deadlock.  Fortunately, the unscientific assumption
> of the
> step (2) makes this rather unprobable.



Yes, agreed this is dangerous and rather unprobable.  I reluctantly agree
with this approach as we have other, more pressing bugs to fix.  Besides, it
is fairly easy to know if we hit problems in this area since the bug will
occur only during shutdown.

Using pthread_cancel() on linux has its own set of advantages and
> disadvantages. By default, threads are created with cancellability enabled
> and
> in "deferred cancellability" state. It means that pthread_cancel() will
> only
> terminate thread when it reaches an explicit cancellation point, such as
> pthread_cond_wait(). In this way, pthread_cancel() does not suffer from
> the
> malloc-deadlock issue.
>
> However, deferred cancellability brings its own problems, for example, as
> pthread_mutex_lock is not a cancellation point, a pair of deadlocked
> daemon
> threads will still be alive during VM shutdown, and could probably result
> in
> EINVAL return and subsequent assertion failure if the monitors are
> destroyed in
> the main shutdown thread.
>
> So, the shutdown algorithm has been improved compared to the former
> exception-based approach, like Thread.stop(), but still is not perfect.
> Eugene and me discussed this and agreed to keep an open eye on shutdown
> bugs.



The shutdown code has definitely been improved.  If it works for the apps we
care about, then its good enough for the time being.

There is still a field for experiments and possible improvements, e.g. using
> asynchronous cancellability or even using pthread_kill instead of
> pthread_cancel to implement immediate thread termination.



Hmm... resource reclaimation is always the worst part of building systems
software.  It will probably take a few iterations to get it right.  Somehow
Linux can core dump a bad process and successfully recover all dangling file
handles, network sockets, page directories, thread info blocks, etc.  Maybe
we look at how OS's do resource recovery for some ideas.

--
> Salikh Zakirov
>
>


-- 
Weldon Washburn
Intel Enterprise Solutions Software Division

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Evgueni Brevnov <ev...@gmail.com>]

On 3/23/07, Rana Dasgupta <rd...@gmail.com> wrote:
> I am not sure what the current thinking is on this, but I would suggest the
> opposite to 3289 ... tying the lifetime of the hythread structure to the
> java thread object and storing a reference to it in the thread object. When
> the OS thread goes away the association between its TLS and the hythread
> structure goes away, but the structure itself is preserved till GC collects
> the java thread object when the hythread memory block will need to be
> released back. We should also store all thread state unambiguously in the
> hythread structure (and not split across the two locations ) and protect
> updates to it with an unmanaged lock. This would ensure that all api's that
> read/update thread state, jvmti, any cycle detection code we may add will
> look in one place only. That is easier said than done, but it will be nice
> to move in this direction.
>
> I tried out various shutdown sequences in the debugger and they all seemed
> to work. So though terminatethread is about the worst thing we can do on
> Windows, the sequence below seems to work. I noticed that while we prevent
> races in vm_destroy() when multiple threads could call System.exit()
> simultaneously with an APR global lock, there is nothing preventing new
> threads from being created while shutdown is in progress.

Actually, there is a patch which contains a soultion for the problem
of new threads creation during VM shutdown. Unfortunately, it was
decided not to commit this patch as useless. See
https://issues.apache.org/jira/browse/HARMONY-2858

Thanks
Evgueni

>
> Less important, I also noticed that we do nothing with dll attach/detach
> notifications. If we use 10 dll's in every java process and each dllmain
> touches 2 pages( 1 data, 1 code ), that's 20 pages added to the working
> set at startup. That's about a 100K. We can just DisableThreadLibraryCalls()
> to switch off these notifications.
>
> Thanks,
> Rana
>
>
>
>
> On 3/16/07, Salikh Zakirov <Sa...@intel.com> wrote:
> >
> > Weldon Washburn wrote:
> > > On another topic, do we really have to use Thread.stop() in the
> > > implementation of vm_shutdown_stop_java_threads() ?  This seems way
> > > too hard.   All we really want to do is give back all the OS
> > resources.  We
> > > could simply call OS kill on each specific thread, return all memory via
> > > "free()", return all file handles, etc.  Note if there was no need to
> > allow
> > > multiple JVMs in a single address space to come and go, we could
> > simplify
> > > the above and basically just call process exit().
> >
> > HARMONY-3381 (already committed) replaces Thread.stop-like shutdown
> > algorithm
> > to the following one
> >
> > 1) set a shutdown callback, which executes hythread_exit() (which
> > delegates to
> > either pthread_exit() or ExitThread())
> >
> > 2) wait for a little while -- this is a little bit "unscientific" and
> > assumes,
> > that all actively running threads will reach a callback safepoint within
> > that
> > "little while". If assumption holds true, the remaining threads are highly
> > likely to be unactive threads, i.e. sleeping, waiting or blocked.
> > The key about a little while is that if all threads has terminated
> > promptly, we
> > don't need to wait any longer at all. It is currently implemented using
> > hythread_join() with timeout.
> >
> > 3) on the third step the remaining threads are terminated using
> > hythread_cancel() (which delegates to either of pthread_cancel() or
> > TerminateThread()) -- This step is inherently dangerous on Windows, as
> > terminated thread may be holding system-wide malloc lock, and subsequent
> > free()
> > on a main thread will deadlock. Fortunately, the unscientific assumption
> > of the
> > step (2) makes this rather unprobable.
> >
> > Using pthread_cancel() on linux has its own set of advantages and
> > disadvantages. By default, threads are created with cancellability enabled
> > and
> > in "deferred cancellability" state. It means that pthread_cancel() will
> > only
> > terminate thread when it reaches an explicit cancellation point, such as
> > pthread_cond_wait(). In this way, pthread_cancel() does not suffer from
> > the
> > malloc-deadlock issue.
> >
> > However, deferred cancellability brings its own problems, for example, as
> > pthread_mutex_lock is not a cancellation point, a pair of deadlocked
> > daemon
> > threads will still be alive during VM shutdown, and could probably result
> > in
> > EINVAL return and subsequent assertion failure if the monitors are
> > destroyed in
> > the main shutdown thread.
> >
> > So, the shutdown algorithm has been improved compared to the former
> > exception-based approach, like Thread.stop(), but still is not perfect.
> > Eugene and me discussed this and agreed to keep an open eye on shutdown
> > bugs.
> > There is still a field for experiments and possible improvements, e.g.
> > using
> > asynchronous cancellability or even using pthread_kill instead of
> > pthread_cancel to implement immediate thread termination.
> >
> > --
> > Salikh Zakirov
> >
> >
>

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Rana Dasgupta <rd...@gmail.com>]

I am not sure what the current thinking is on this, but I would suggest the
opposite to 3289 ... tying the lifetime of the hythread structure to the
java thread object and storing a reference to it in the thread object. When
the OS thread goes away the association between its TLS and the hythread
structure goes away, but the structure itself is preserved till GC collects
the java thread object when the hythread memory block will need to be
released back. We should also store all thread state unambiguously in the
hythread structure (and not split across the two locations ) and protect
updates to it with an unmanaged lock. This would ensure that all api's that
read/update thread state, jvmti, any cycle detection code we may add will
look in one place only. That is easier said than done, but it will be nice
to move in this direction.

I tried out various shutdown sequences in the debugger and they all seemed
to work. So though terminatethread is about the worst thing we can do on
Windows, the sequence below seems to work. I noticed that while we prevent
races in vm_destroy() when multiple threads could call System.exit()
simultaneously with an APR global lock, there is nothing preventing new
threads from being created while shutdown is in progress.

Less important, I also noticed that we do nothing with dll attach/detach
notifications. If we use 10 dll's in every java process and each dllmain
touches 2 pages( 1 data, 1 code ), that's 20 pages added to the working
set at startup. That's about a 100K. We can just DisableThreadLibraryCalls()
to switch off these notifications.

Thanks,
Rana




On 3/16/07, Salikh Zakirov <Sa...@intel.com> wrote:
>
> Weldon Washburn wrote:
> > On another topic, do we really have to use Thread.stop() in the
> > implementation of vm_shutdown_stop_java_threads() ?  This seems way
> > too hard.   All we really want to do is give back all the OS
> resources.  We
> > could simply call OS kill on each specific thread, return all memory via
> > "free()", return all file handles, etc.  Note if there was no need to
> allow
> > multiple JVMs in a single address space to come and go, we could
> simplify
> > the above and basically just call process exit().
>
> HARMONY-3381 (already committed) replaces Thread.stop-like shutdown
> algorithm
> to the following one
>
> 1) set a shutdown callback, which executes hythread_exit() (which
> delegates to
> either pthread_exit() or ExitThread())
>
> 2) wait for a little while -- this is a little bit "unscientific" and
> assumes,
> that all actively running threads will reach a callback safepoint within
> that
> "little while". If assumption holds true, the remaining threads are highly
> likely to be unactive threads, i.e. sleeping, waiting or blocked.
> The key about a little while is that if all threads has terminated
> promptly, we
> don't need to wait any longer at all. It is currently implemented using
> hythread_join() with timeout.
>
> 3) on the third step the remaining threads are terminated using
> hythread_cancel() (which delegates to either of pthread_cancel() or
> TerminateThread()) -- This step is inherently dangerous on Windows, as
> terminated thread may be holding system-wide malloc lock, and subsequent
> free()
> on a main thread will deadlock. Fortunately, the unscientific assumption
> of the
> step (2) makes this rather unprobable.
>
> Using pthread_cancel() on linux has its own set of advantages and
> disadvantages. By default, threads are created with cancellability enabled
> and
> in "deferred cancellability" state. It means that pthread_cancel() will
> only
> terminate thread when it reaches an explicit cancellation point, such as
> pthread_cond_wait(). In this way, pthread_cancel() does not suffer from
> the
> malloc-deadlock issue.
>
> However, deferred cancellability brings its own problems, for example, as
> pthread_mutex_lock is not a cancellation point, a pair of deadlocked
> daemon
> threads will still be alive during VM shutdown, and could probably result
> in
> EINVAL return and subsequent assertion failure if the monitors are
> destroyed in
> the main shutdown thread.
>
> So, the shutdown algorithm has been improved compared to the former
> exception-based approach, like Thread.stop(), but still is not perfect.
> Eugene and me discussed this and agreed to keep an open eye on shutdown
> bugs.
> There is still a field for experiments and possible improvements, e.g.
> using
> asynchronous cancellability or even using pthread_kill instead of
> pthread_cancel to implement immediate thread termination.
>
> --
> Salikh Zakirov
>
>

Re: [drlvm][threading] H3289 -- some JVMTI questions

[Salikh Zakirov <Sa...@Intel.com>]

Weldon Washburn wrote:
> On another topic, do we really have to use Thread.stop() in the
> implementation of vm_shutdown_stop_java_threads() ?  This seems way
> too hard.   All we really want to do is give back all the OS resources.  We
> could simply call OS kill on each specific thread, return all memory via
> "free()", return all file handles, etc.  Note if there was no need to allow
> multiple JVMs in a single address space to come and go, we could simplify
> the above and basically just call process exit().

HARMONY-3381 (already committed) replaces Thread.stop-like shutdown algorithm
to the following one

1) set a shutdown callback, which executes hythread_exit() (which delegates to
either pthread_exit() or ExitThread())

2) wait for a little while -- this is a little bit "unscientific" and assumes,
that all actively running threads will reach a callback safepoint within that
"little while". If assumption holds true, the remaining threads are highly
likely to be unactive threads, i.e. sleeping, waiting or blocked.
The key about a little while is that if all threads has terminated promptly, we
don't need to wait any longer at all. It is currently implemented using
hythread_join() with timeout.

3) on the third step the remaining threads are terminated using
hythread_cancel() (which delegates to either of pthread_cancel() or
TerminateThread()) -- This step is inherently dangerous on Windows, as
terminated thread may be holding system-wide malloc lock, and subsequent free()
on a main thread will deadlock. Fortunately, the unscientific assumption of the
step (2) makes this rather unprobable.

Using pthread_cancel() on linux has its own set of advantages and
disadvantages. By default, threads are created with cancellability enabled and
in "deferred cancellability" state. It means that pthread_cancel() will only
terminate thread when it reaches an explicit cancellation point, such as
pthread_cond_wait(). In this way, pthread_cancel() does not suffer from the
malloc-deadlock issue.

However, deferred cancellability brings its own problems, for example, as
pthread_mutex_lock is not a cancellation point, a pair of deadlocked daemon
threads will still be alive during VM shutdown, and could probably result in
EINVAL return and subsequent assertion failure if the monitors are destroyed in
the main shutdown thread.

So, the shutdown algorithm has been improved compared to the former
exception-based approach, like Thread.stop(), but still is not perfect.
Eugene and me discussed this and agreed to keep an open eye on shutdown bugs.
There is still a field for experiments and possible improvements, e.g. using
asynchronous cancellability or even using pthread_kill instead of
pthread_cancel to implement immediate thread termination.

-- 
Salikh Zakirov