[drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

Hi All,

I'd like to share with you my findings regarding current DRLVM
shutdown procedure (not in too many details) and discuss further steps
to improve it.

/* General words about shutdown */

Here is a snip from the j.l.Runtime.addShutdownHook() spec:

"The Java virtual machine shuts down in response to two kinds of events:
1) The program exits normally, when the last non-daemon thread exits
or when the exit (equivalently, System.exit) method is invoked, or
2) The virtual machine is terminated in response to a user interrupt,
such as typing ^C, or a system-wide event, such as user logoff or
system shutdown."

Except these 2 events there is one more player on the scene. It is
DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
request reclamation of  all resources owned by the VM.

As it was discussed earlier (and I think agreed) the System.exit()
should terminate entire process the VM is running in right after it
executed shutdown hooks and finalizes. At least RI does so. Thus I
would like to look closer at the case when the last non-daemon thread
exits and the VM is requested to cleanup resources through
DestroyJavaVM. When DestroyJavaVM is called it waits until current
thread is the last non-daemon thread before cleaning up all resources
owned by the VM. It is important to note it is still possible daemon
java and native threads are running inside the VM even if there is no
running non-daemon threads.  These running threads may still be using
VM-wide data (generally located in Global_Env). Thus before cleaning
resources up we need to stop these threads somehow. Unfortunately, we
can't just kill threads. We need to release all resources (system
locks, heap/stack memory, open files/connections, etc) owed by each
thread. So the problem I'd like to discuss and solve: How to stop
running threads in a safe manner?

/* Current DRLVM implementation */

Currently, the DRLVM isn't able to stop running threads in a safe way.
Thus it just doesn't clean resources up if there are running threads
after all non-daemon threads exits.

/* How to stop threads safely. Proposal. */

I propose to consider native and java threads separately. Lets start
with native threads.

Native threads: It seems there is not much we can do to stop arbitrary
native thread and release resources owed by it. I don't consider
global resource management as a possible solution here :-). Thus there
are two requirements for native threads to ensure safe VM shutdown:

a) Any native thread must be created by TM or attached to TM. So TM is
aware about all running native threads.
b) Native thread provides a callback which is called to request thread
shutdown. It's the native thread responsibility to clean its resources
upon shutdown request.

Java threads: We have much more control other java threads. Namely it
is possible to unwind the stack in a safe way. When the VM needs to
stop a java thread it asynchronously raises an exception to that
thread. The VM ensures that the exception will not be caught by any
java exception handler (ala j.l.ThreadDeath exception). This guarantee
full and safe java stack unwinding. If there is no user native frames
on the stack (JNI calls) then thread exits normally otherwise the
control is returned to the user's code with an exception raised. It is
the user's responsibility to handle exception properly and clean up
all resources allocated by native code.

Any suggestions/comments are HIGHLY welcome.

Thanks
Evgueni

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Tim Ellison <t....@gmail.com> wrote:
> Evgueni Brevnov wrote:
> > On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> >> Evgueni, when the process is stopped, why do we need care about the
> >> system resource in details if they are released by the system
> >> automatically?
> >
> > Xiao-Feng, the process is not necessarily stopped. The invocation API
> > defines standard way to create/destroy VM instances. Thus VM
> > destroying doesn't mean process termination.
>
> Absolutely, the goal should be to be able to create and destroy VM's in
> a long running loop without leaking resources.

Agreed.

>
> >> And the requirement below for all the native threads to comply with TM
> >> rule sounds like unrealistic.
> >
> > It is good that you asked this question. (So you read my proposal
> > more-or-less carefully :-)). Let me describe this part in more
> > details. There are two different kind of native threads. Lets call
> > them user and helper threads. The difference is that user threads are
> > created by the application native code while helper threads are
> > created by VM components. There is no chance for a user thread to
> > interact with VM until the thread is not attached to VM. So there is
> > no need to care about such threads. (User should care by itself :-))
>
> Agreed.  If the thread has no interaction with the VM, say it is created
> by a third-party library that knows nothing of the VM, then we cannot
> help and it must clean up itself.  However, the resource management API
> we define should be generic, so we can register a callback to such
> third-party libraries if they have a clean-up function we need to call.

Agreed.

>
> > The helper threads are created by VM components and may access VM data
> > through inter-component interfaces. Thus we need to be sure that no
> > such threads are running before destroying VM-wide data.
>
> Yep, that is part of the shutdown sequence.  As I wrote elsewhere we
> should describe the state of the VM data at the point the callbacks are
> invoked, and limit the operations that are permissible in the callback
> function, e.g. creating a new Thread to run arbitrary Java code is
> likely disallowed <g>.

Agreed.

Evgueni

>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Xiao-Feng Li <xi...@gmail.com>]

On 11/21/06, Tim Ellison <t....@gmail.com> wrote:
> Evgueni Brevnov wrote:
> > On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> >> Evgueni, when the process is stopped, why do we need care about the
> >> system resource in details if they are released by the system
> >> automatically?
> >
> > Xiao-Feng, the process is not necessarily stopped. The invocation API
> > defines standard way to create/destroy VM instances. Thus VM
> > destroying doesn't mean process termination.
>
> Absolutely, the goal should be to be able to create and destroy VM's in
> a long running loop without leaking resources.
>
> >> And the requirement below for all the native threads to comply with TM
> >> rule sounds like unrealistic.
> >
> > It is good that you asked this question. (So you read my proposal
> > more-or-less carefully :-)). Let me describe this part in more
> > details. There are two different kind of native threads. Lets call
> > them user and helper threads. The difference is that user threads are
> > created by the application native code while helper threads are
> > created by VM components. There is no chance for a user thread to
> > interact with VM until the thread is not attached to VM. So there is
> > no need to care about such threads. (User should care by itself :-))
>
> Agreed.  If the thread has no interaction with the VM, say it is created
> by a third-party library that knows nothing of the VM, then we cannot
> help and it must clean up itself.  However, the resource management API
> we define should be generic, so we can register a callback to such
> third-party libraries if they have a clean-up function we need to call.
>
> > The helper threads are created by VM components and may access VM data
> > through inter-component interfaces. Thus we need to be sure that no
> > such threads are running before destroying VM-wide data.
>
> Yep, that is part of the shutdown sequence.  As I wrote elsewhere we
> should describe the state of the VM data at the point the callbacks are
> invoked, and limit the operations that are permissible in the callback
> function, e.g. creating a new Thread to run arbitrary Java code is
> likely disallowed <g>.

Agreed, and I think not all the restrictions can be automatically
checked and enforced.

Thanks,
xiaofeng

> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Evgueni Brevnov wrote:
> On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
>> Evgueni, when the process is stopped, why do we need care about the
>> system resource in details if they are released by the system
>> automatically?
> 
> Xiao-Feng, the process is not necessarily stopped. The invocation API
> defines standard way to create/destroy VM instances. Thus VM
> destroying doesn't mean process termination.

Absolutely, the goal should be to be able to create and destroy VM's in
a long running loop without leaking resources.

>> And the requirement below for all the native threads to comply with TM
>> rule sounds like unrealistic.
> 
> It is good that you asked this question. (So you read my proposal
> more-or-less carefully :-)). Let me describe this part in more
> details. There are two different kind of native threads. Lets call
> them user and helper threads. The difference is that user threads are
> created by the application native code while helper threads are
> created by VM components. There is no chance for a user thread to
> interact with VM until the thread is not attached to VM. So there is
> no need to care about such threads. (User should care by itself :-))

Agreed.  If the thread has no interaction with the VM, say it is created
by a third-party library that knows nothing of the VM, then we cannot
help and it must clean up itself.  However, the resource management API
we define should be generic, so we can register a callback to such
third-party libraries if they have a clean-up function we need to call.

> The helper threads are created by VM components and may access VM data
> through inter-component interfaces. Thus we need to be sure that no
> such threads are running before destroying VM-wide data.

Yep, that is part of the shutdown sequence.  As I wrote elsewhere we
should describe the state of the VM data at the point the callbacks are
invoked, and limit the operations that are permissible in the callback
function, e.g. creating a new Thread to run arbitrary Java code is
likely disallowed <g>.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> On 11/21/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> > On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> > > On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> > > > Hi All,
> > > >
> > > > I'd like to share with you my findings regarding current DRLVM
> > > > shutdown procedure (not in too many details) and discuss further steps
> > > > to improve it.
> > > >
> > > > /* General words about shutdown */
> > > >
> > > > Here is a snip from the j.l.Runtime.addShutdownHook() spec:
> > > >
> > > > "The Java virtual machine shuts down in response to two kinds of events:
> > > > 1) The program exits normally, when the last non-daemon thread exits
> > > > or when the exit (equivalently, System.exit) method is invoked, or
> > > > 2) The virtual machine is terminated in response to a user interrupt,
> > > > such as typing ^C, or a system-wide event, such as user logoff or
> > > > system shutdown."
> > > >
> > > > Except these 2 events there is one more player on the scene. It is
> > > > DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> > > > request reclamation of  all resources owned by the VM.
> > > >
> > > > As it was discussed earlier (and I think agreed) the System.exit()
> > > > should terminate entire process the VM is running in right after it
> > > > executed shutdown hooks and finalizes. At least RI does so. Thus I
> > > > would like to look closer at the case when the last non-daemon thread
> > > > exits and the VM is requested to cleanup resources through
> > > > DestroyJavaVM. When DestroyJavaVM is called it waits until current
> > > > thread is the last non-daemon thread before cleaning up all resources
> > > > owned by the VM. It is important to note it is still possible daemon
> > > > java and native threads are running inside the VM even if there is no
> > > > running non-daemon threads.  These running threads may still be using
> > > > VM-wide data (generally located in Global_Env). Thus before cleaning
> > > > resources up we need to stop these threads somehow. Unfortunately, we
> > > > can't just kill threads. We need to release all resources (system
> > > > locks, heap/stack memory, open files/connections, etc) owed by each
> > > > thread. So the problem I'd like to discuss and solve: How to stop
> > > > running threads in a safe manner?
> > >
> > > Evgueni, when the process is stopped, why do we need care about the
> > > system resource in details if they are released by the system
> > > automatically?
> >
> > Xiao-Feng, the process is not necessarily stopped. The invocation API
> > defines standard way to create/destroy VM instances. Thus VM
> > destroying doesn't mean process termination.
> >
> > >
> > > And the requirement below for all the native threads to comply with TM
> > > rule sounds like unrealistic.
> >
> > It is good that you asked this question. (So you read my proposal
> > more-or-less carefully :-)). Let me describe this part in more
> > details. There are two different kind of native threads. Lets call
> > them user and helper threads. The difference is that user threads are
> > created by the application native code while helper threads are
> > created by VM components. There is no chance for a user thread to
> > interact with VM until the thread is not attached to VM. So there is
> > no need to care about such threads. (User should care by itself :-))
> > The helper threads are created by VM components and may access VM data
> > through inter-component interfaces. Thus we need to be sure that no
> > such threads are running before destroying VM-wide data.
>
> I am not sure if this difference really makes much difference. I think
> for resource management, some coding conventions should better be
> encouraged, just like object finalizer. We can't guarantee everything
> can happen automatically and correctly.

In case of native threads we just guarantee to call registered
callback functions. Nothing more...

>
> > Thanks
> > Evgueni
> > >
> > > Thanks,
> > > xiaofeng
> > >
> > > > /* Current DRLVM implementation */
> > > >
> > > > Currently, the DRLVM isn't able to stop running threads in a safe way.
> > > > Thus it just doesn't clean resources up if there are running threads
> > > > after all non-daemon threads exits.
> > > >
> > > > /* How to stop threads safely. Proposal. */
> > > >
> > > > I propose to consider native and java threads separately. Lets start
> > > > with native threads.
> > > >
> > > > Native threads: It seems there is not much we can do to stop arbitrary
> > > > native thread and release resources owed by it. I don't consider
> > > > global resource management as a possible solution here :-). Thus there
> > > > are two requirements for native threads to ensure safe VM shutdown:
> > > >
> > > > a) Any native thread must be created by TM or attached to TM. So TM is
> > > > aware about all running native threads.
> > > > b) Native thread provides a callback which is called to request thread
> > > > shutdown. It's the native thread responsibility to clean its resources
> > > > upon shutdown request.
> > > >
> > > > Java threads: We have much more control other java threads. Namely it
> > > > is possible to unwind the stack in a safe way. When the VM needs to
> > > > stop a java thread it asynchronously raises an exception to that
> > > > thread. The VM ensures that the exception will not be caught by any
> > > > java exception handler (ala j.l.ThreadDeath exception). This guarantee
> > > > full and safe java stack unwinding. If there is no user native frames
> > > > on the stack (JNI calls) then thread exits normally otherwise the
> > > > control is returned to the user's code with an exception raised. It is
> > > > the user's responsibility to handle exception properly and clean up
> > > > all resources allocated by native code.
> > > >
> > > > Any suggestions/comments are HIGHLY welcome.
> > > >
> > > > Thanks
> > > > Evgueni
> > > >
> > >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Xiao-Feng Li <xi...@gmail.com>]

On 11/21/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> > On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> > > Hi All,
> > >
> > > I'd like to share with you my findings regarding current DRLVM
> > > shutdown procedure (not in too many details) and discuss further steps
> > > to improve it.
> > >
> > > /* General words about shutdown */
> > >
> > > Here is a snip from the j.l.Runtime.addShutdownHook() spec:
> > >
> > > "The Java virtual machine shuts down in response to two kinds of events:
> > > 1) The program exits normally, when the last non-daemon thread exits
> > > or when the exit (equivalently, System.exit) method is invoked, or
> > > 2) The virtual machine is terminated in response to a user interrupt,
> > > such as typing ^C, or a system-wide event, such as user logoff or
> > > system shutdown."
> > >
> > > Except these 2 events there is one more player on the scene. It is
> > > DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> > > request reclamation of  all resources owned by the VM.
> > >
> > > As it was discussed earlier (and I think agreed) the System.exit()
> > > should terminate entire process the VM is running in right after it
> > > executed shutdown hooks and finalizes. At least RI does so. Thus I
> > > would like to look closer at the case when the last non-daemon thread
> > > exits and the VM is requested to cleanup resources through
> > > DestroyJavaVM. When DestroyJavaVM is called it waits until current
> > > thread is the last non-daemon thread before cleaning up all resources
> > > owned by the VM. It is important to note it is still possible daemon
> > > java and native threads are running inside the VM even if there is no
> > > running non-daemon threads.  These running threads may still be using
> > > VM-wide data (generally located in Global_Env). Thus before cleaning
> > > resources up we need to stop these threads somehow. Unfortunately, we
> > > can't just kill threads. We need to release all resources (system
> > > locks, heap/stack memory, open files/connections, etc) owed by each
> > > thread. So the problem I'd like to discuss and solve: How to stop
> > > running threads in a safe manner?
> >
> > Evgueni, when the process is stopped, why do we need care about the
> > system resource in details if they are released by the system
> > automatically?
>
> Xiao-Feng, the process is not necessarily stopped. The invocation API
> defines standard way to create/destroy VM instances. Thus VM
> destroying doesn't mean process termination.
>
> >
> > And the requirement below for all the native threads to comply with TM
> > rule sounds like unrealistic.
>
> It is good that you asked this question. (So you read my proposal
> more-or-less carefully :-)). Let me describe this part in more
> details. There are two different kind of native threads. Lets call
> them user and helper threads. The difference is that user threads are
> created by the application native code while helper threads are
> created by VM components. There is no chance for a user thread to
> interact with VM until the thread is not attached to VM. So there is
> no need to care about such threads. (User should care by itself :-))
> The helper threads are created by VM components and may access VM data
> through inter-component interfaces. Thus we need to be sure that no
> such threads are running before destroying VM-wide data.

I am not sure if this difference really makes much difference. I think
for resource management, some coding conventions should better be
encouraged, just like object finalizer. We can't guarantee everything
can happen automatically and correctly.

> Thanks
> Evgueni
> >
> > Thanks,
> > xiaofeng
> >
> > > /* Current DRLVM implementation */
> > >
> > > Currently, the DRLVM isn't able to stop running threads in a safe way.
> > > Thus it just doesn't clean resources up if there are running threads
> > > after all non-daemon threads exits.
> > >
> > > /* How to stop threads safely. Proposal. */
> > >
> > > I propose to consider native and java threads separately. Lets start
> > > with native threads.
> > >
> > > Native threads: It seems there is not much we can do to stop arbitrary
> > > native thread and release resources owed by it. I don't consider
> > > global resource management as a possible solution here :-). Thus there
> > > are two requirements for native threads to ensure safe VM shutdown:
> > >
> > > a) Any native thread must be created by TM or attached to TM. So TM is
> > > aware about all running native threads.
> > > b) Native thread provides a callback which is called to request thread
> > > shutdown. It's the native thread responsibility to clean its resources
> > > upon shutdown request.
> > >
> > > Java threads: We have much more control other java threads. Namely it
> > > is possible to unwind the stack in a safe way. When the VM needs to
> > > stop a java thread it asynchronously raises an exception to that
> > > thread. The VM ensures that the exception will not be caught by any
> > > java exception handler (ala j.l.ThreadDeath exception). This guarantee
> > > full and safe java stack unwinding. If there is no user native frames
> > > on the stack (JNI calls) then thread exits normally otherwise the
> > > control is returned to the user's code with an exception raised. It is
> > > the user's responsibility to handle exception properly and clean up
> > > all resources allocated by native code.
> > >
> > > Any suggestions/comments are HIGHLY welcome.
> > >
> > > Thanks
> > > Evgueni
> > >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Gregory Shimansky <gs...@gmail.com> wrote:
> Evgueni Brevnov wrote:
> > On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> >> On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> >> > Hi All,
> >> >
> >> > I'd like to share with you my findings regarding current DRLVM
> >> > shutdown procedure (not in too many details) and discuss further steps
> >> > to improve it.
> >> >
> >> > /* General words about shutdown */
> >> >
> >> > Here is a snip from the j.l.Runtime.addShutdownHook() spec:
> >> >
> >> > "The Java virtual machine shuts down in response to two kinds of
> >> events:
> >> > 1) The program exits normally, when the last non-daemon thread exits
> >> > or when the exit (equivalently, System.exit) method is invoked, or
> >> > 2) The virtual machine is terminated in response to a user interrupt,
> >> > such as typing ^C, or a system-wide event, such as user logoff or
> >> > system shutdown."
> >> >
> >> > Except these 2 events there is one more player on the scene. It is
> >> > DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> >> > request reclamation of  all resources owned by the VM.
> >> >
> >> > As it was discussed earlier (and I think agreed) the System.exit()
> >> > should terminate entire process the VM is running in right after it
> >> > executed shutdown hooks and finalizes. At least RI does so. Thus I
> >> > would like to look closer at the case when the last non-daemon thread
> >> > exits and the VM is requested to cleanup resources through
> >> > DestroyJavaVM. When DestroyJavaVM is called it waits until current
> >> > thread is the last non-daemon thread before cleaning up all resources
> >> > owned by the VM. It is important to note it is still possible daemon
> >> > java and native threads are running inside the VM even if there is no
> >> > running non-daemon threads.  These running threads may still be using
> >> > VM-wide data (generally located in Global_Env). Thus before cleaning
> >> > resources up we need to stop these threads somehow. Unfortunately, we
> >> > can't just kill threads. We need to release all resources (system
> >> > locks, heap/stack memory, open files/connections, etc) owed by each
> >> > thread. So the problem I'd like to discuss and solve: How to stop
> >> > running threads in a safe manner?
> >>
> >> Evgueni, when the process is stopped, why do we need care about the
> >> system resource in details if they are released by the system
> >> automatically?
> >
> > Xiao-Feng, the process is not necessarily stopped. The invocation API
> > defines standard way to create/destroy VM instances. Thus VM
> > destroying doesn't mean process termination.
>
> I want to add my 2 cents. Resources cleanup is also useful for analyzing
> memory leaks. If VM and classlib don't release their resources on
> shutdown, any memory leak analyzers would complain a lot about memory
> which wasn't released.

Yes, exactly. I was thinking about looping VM creation/destroying
operations as a good test to detect resource leaking.

Evgueni
>
> --
> Gregory
>
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Gregory Shimansky <gs...@gmail.com>]

Evgueni Brevnov wrote:
> On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
>> On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I'd like to share with you my findings regarding current DRLVM
>> > shutdown procedure (not in too many details) and discuss further steps
>> > to improve it.
>> >
>> > /* General words about shutdown */
>> >
>> > Here is a snip from the j.l.Runtime.addShutdownHook() spec:
>> >
>> > "The Java virtual machine shuts down in response to two kinds of 
>> events:
>> > 1) The program exits normally, when the last non-daemon thread exits
>> > or when the exit (equivalently, System.exit) method is invoked, or
>> > 2) The virtual machine is terminated in response to a user interrupt,
>> > such as typing ^C, or a system-wide event, such as user logoff or
>> > system shutdown."
>> >
>> > Except these 2 events there is one more player on the scene. It is
>> > DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
>> > request reclamation of  all resources owned by the VM.
>> >
>> > As it was discussed earlier (and I think agreed) the System.exit()
>> > should terminate entire process the VM is running in right after it
>> > executed shutdown hooks and finalizes. At least RI does so. Thus I
>> > would like to look closer at the case when the last non-daemon thread
>> > exits and the VM is requested to cleanup resources through
>> > DestroyJavaVM. When DestroyJavaVM is called it waits until current
>> > thread is the last non-daemon thread before cleaning up all resources
>> > owned by the VM. It is important to note it is still possible daemon
>> > java and native threads are running inside the VM even if there is no
>> > running non-daemon threads.  These running threads may still be using
>> > VM-wide data (generally located in Global_Env). Thus before cleaning
>> > resources up we need to stop these threads somehow. Unfortunately, we
>> > can't just kill threads. We need to release all resources (system
>> > locks, heap/stack memory, open files/connections, etc) owed by each
>> > thread. So the problem I'd like to discuss and solve: How to stop
>> > running threads in a safe manner?
>>
>> Evgueni, when the process is stopped, why do we need care about the
>> system resource in details if they are released by the system
>> automatically?
> 
> Xiao-Feng, the process is not necessarily stopped. The invocation API
> defines standard way to create/destroy VM instances. Thus VM
> destroying doesn't mean process termination.

I want to add my 2 cents. Resources cleanup is also useful for analyzing 
memory leaks. If VM and classlib don't release their resources on 
shutdown, any memory leak analyzers would complain a lot about memory 
which wasn't released.

-- 
Gregory

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Xiao-Feng Li <xi...@gmail.com> wrote:
> On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> > Hi All,
> >
> > I'd like to share with you my findings regarding current DRLVM
> > shutdown procedure (not in too many details) and discuss further steps
> > to improve it.
> >
> > /* General words about shutdown */
> >
> > Here is a snip from the j.l.Runtime.addShutdownHook() spec:
> >
> > "The Java virtual machine shuts down in response to two kinds of events:
> > 1) The program exits normally, when the last non-daemon thread exits
> > or when the exit (equivalently, System.exit) method is invoked, or
> > 2) The virtual machine is terminated in response to a user interrupt,
> > such as typing ^C, or a system-wide event, such as user logoff or
> > system shutdown."
> >
> > Except these 2 events there is one more player on the scene. It is
> > DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> > request reclamation of  all resources owned by the VM.
> >
> > As it was discussed earlier (and I think agreed) the System.exit()
> > should terminate entire process the VM is running in right after it
> > executed shutdown hooks and finalizes. At least RI does so. Thus I
> > would like to look closer at the case when the last non-daemon thread
> > exits and the VM is requested to cleanup resources through
> > DestroyJavaVM. When DestroyJavaVM is called it waits until current
> > thread is the last non-daemon thread before cleaning up all resources
> > owned by the VM. It is important to note it is still possible daemon
> > java and native threads are running inside the VM even if there is no
> > running non-daemon threads.  These running threads may still be using
> > VM-wide data (generally located in Global_Env). Thus before cleaning
> > resources up we need to stop these threads somehow. Unfortunately, we
> > can't just kill threads. We need to release all resources (system
> > locks, heap/stack memory, open files/connections, etc) owed by each
> > thread. So the problem I'd like to discuss and solve: How to stop
> > running threads in a safe manner?
>
> Evgueni, when the process is stopped, why do we need care about the
> system resource in details if they are released by the system
> automatically?

Xiao-Feng, the process is not necessarily stopped. The invocation API
defines standard way to create/destroy VM instances. Thus VM
destroying doesn't mean process termination.

>
> And the requirement below for all the native threads to comply with TM
> rule sounds like unrealistic.

It is good that you asked this question. (So you read my proposal
more-or-less carefully :-)). Let me describe this part in more
details. There are two different kind of native threads. Lets call
them user and helper threads. The difference is that user threads are
created by the application native code while helper threads are
created by VM components. There is no chance for a user thread to
interact with VM until the thread is not attached to VM. So there is
no need to care about such threads. (User should care by itself :-))
The helper threads are created by VM components and may access VM data
through inter-component interfaces. Thus we need to be sure that no
such threads are running before destroying VM-wide data.

Thanks
Evgueni
>
> Thanks,
> xiaofeng
>
> > /* Current DRLVM implementation */
> >
> > Currently, the DRLVM isn't able to stop running threads in a safe way.
> > Thus it just doesn't clean resources up if there are running threads
> > after all non-daemon threads exits.
> >
> > /* How to stop threads safely. Proposal. */
> >
> > I propose to consider native and java threads separately. Lets start
> > with native threads.
> >
> > Native threads: It seems there is not much we can do to stop arbitrary
> > native thread and release resources owed by it. I don't consider
> > global resource management as a possible solution here :-). Thus there
> > are two requirements for native threads to ensure safe VM shutdown:
> >
> > a) Any native thread must be created by TM or attached to TM. So TM is
> > aware about all running native threads.
> > b) Native thread provides a callback which is called to request thread
> > shutdown. It's the native thread responsibility to clean its resources
> > upon shutdown request.
> >
> > Java threads: We have much more control other java threads. Namely it
> > is possible to unwind the stack in a safe way. When the VM needs to
> > stop a java thread it asynchronously raises an exception to that
> > thread. The VM ensures that the exception will not be caught by any
> > java exception handler (ala j.l.ThreadDeath exception). This guarantee
> > full and safe java stack unwinding. If there is no user native frames
> > on the stack (JNI calls) then thread exits normally otherwise the
> > control is returned to the user's code with an exception raised. It is
> > the user's responsibility to handle exception properly and clean up
> > all resources allocated by native code.
> >
> > Any suggestions/comments are HIGHLY welcome.
> >
> > Thanks
> > Evgueni
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Xiao-Feng Li <xi...@gmail.com>]

On 11/20/06, Evgueni Brevnov <ev...@gmail.com> wrote:
> Hi All,
>
> I'd like to share with you my findings regarding current DRLVM
> shutdown procedure (not in too many details) and discuss further steps
> to improve it.
>
> /* General words about shutdown */
>
> Here is a snip from the j.l.Runtime.addShutdownHook() spec:
>
> "The Java virtual machine shuts down in response to two kinds of events:
> 1) The program exits normally, when the last non-daemon thread exits
> or when the exit (equivalently, System.exit) method is invoked, or
> 2) The virtual machine is terminated in response to a user interrupt,
> such as typing ^C, or a system-wide event, such as user logoff or
> system shutdown."
>
> Except these 2 events there is one more player on the scene. It is
> DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> request reclamation of  all resources owned by the VM.
>
> As it was discussed earlier (and I think agreed) the System.exit()
> should terminate entire process the VM is running in right after it
> executed shutdown hooks and finalizes. At least RI does so. Thus I
> would like to look closer at the case when the last non-daemon thread
> exits and the VM is requested to cleanup resources through
> DestroyJavaVM. When DestroyJavaVM is called it waits until current
> thread is the last non-daemon thread before cleaning up all resources
> owned by the VM. It is important to note it is still possible daemon
> java and native threads are running inside the VM even if there is no
> running non-daemon threads.  These running threads may still be using
> VM-wide data (generally located in Global_Env). Thus before cleaning
> resources up we need to stop these threads somehow. Unfortunately, we
> can't just kill threads. We need to release all resources (system
> locks, heap/stack memory, open files/connections, etc) owed by each
> thread. So the problem I'd like to discuss and solve: How to stop
> running threads in a safe manner?

Evgueni, when the process is stopped, why do we need care about the
system resource in details if they are released by the system
automatically?

And the requirement below for all the native threads to comply with TM
rule sounds like unrealistic.

Thanks,
xiaofeng

> /* Current DRLVM implementation */
>
> Currently, the DRLVM isn't able to stop running threads in a safe way.
> Thus it just doesn't clean resources up if there are running threads
> after all non-daemon threads exits.
>
> /* How to stop threads safely. Proposal. */
>
> I propose to consider native and java threads separately. Lets start
> with native threads.
>
> Native threads: It seems there is not much we can do to stop arbitrary
> native thread and release resources owed by it. I don't consider
> global resource management as a possible solution here :-). Thus there
> are two requirements for native threads to ensure safe VM shutdown:
>
> a) Any native thread must be created by TM or attached to TM. So TM is
> aware about all running native threads.
> b) Native thread provides a callback which is called to request thread
> shutdown. It's the native thread responsibility to clean its resources
> upon shutdown request.
>
> Java threads: We have much more control other java threads. Namely it
> is possible to unwind the stack in a safe way. When the VM needs to
> stop a java thread it asynchronously raises an exception to that
> thread. The VM ensures that the exception will not be caught by any
> java exception handler (ala j.l.ThreadDeath exception). This guarantee
> full and safe java stack unwinding. If there is no user native frames
> on the stack (JNI calls) then thread exits normally otherwise the
> control is returned to the user's code with an exception raised. It is
> the user's responsibility to handle exception properly and clean up
> all resources allocated by native code.
>
> Any suggestions/comments are HIGHLY welcome.
>
> Thanks
> Evgueni
>

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

nice!

Evgueni Brevnov wrote:
> Hi,
> 
> I've implemented safe-shutdown of java threads as it was at the
> beginning of this thread. No significant problems were encountered
> while implementing. Find the patch at
> http://issues.apache.org/jira/browse/HARMONY-2391. Here is the
> details:
> 
> 1) Java thread stack unwinding in shutdown mode implemented.
> 2) Proper processing of SIGINT (Ctrl+C) implemented.
> 3) Proper processing of SIGQUIT (Ctrl+\) implemented.
> 4) JNI function FatalError reimplemented.
> 5) Smoke test infrastructure extented to compile native sources.
> 6) New smoke tests were added to test VM shutdown.
> 
> Note: Enjoy pressing Ctrl+Break to see java stack traces :-) (Ctrl+\ on 
> Linux)
> 
> Thanks
> Evgueni
> 
> On 11/22/06, Alexey Varlamov <al...@gmail.com> wrote:
>> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
>> > Alexey Varlamov wrote:
>> > > 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
>> > >> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
>> > >> > 2006/11/21, Tim Ellison <t....@gmail.com>:
>> > >> > > Evgueni Brevnov wrote:
>> > >> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>> > >> > > >>
>> > >> > > >> Evgueni Brevnov wrote:
>> > >> > > >> >
>> > >> > > >> > As it was discussed earlier (and I think agreed) the
>> > >> System.exit()
>> > >> > > >> > should terminate entire process the VM is running in right
>> > >> after it
>> > >> > > >> > executed shutdown hooks and finalizes.
>> > >> > > >>
>> > >> > > >> I don't think so.  How could you embed the VM in another 
>> program?
>> > >> > > >
>> > >> > > > My initial understanding was that System.exit() shouldn't 
>> terminate
>> > >> > > > the process. But the experiments proved the opposite. AFAIU 
>> if you
>> > >> > > > want to embed the application then it should not call
>> > >> System.exit().
>> > >> > >
>> > >> > > I was surprised to see the results of you experiment too -- 
>> I'd have
>> > >> > > failed that Java pop-quiz question!
>> > >> > >
>> > >> > > > I don't see any problems to extend the proposal to the case of
>> > >> > > > System.exit() so that the process will not be stopped. But, do
>> > >> we want
>> > >> > > > such incompatibility with RI?
>> > >> > >
>> > >> > > No, we can't change that -- if people expect the program to 
>> exit()
>> > >> it we
>> > >> > > will see hangs everywhere.  As somebody (I forget) wrote at the
>> > >> time, it
>> > >> > > becomes the responsibility of the app to set the security 
>> policy to
>> > >> > > disallow exit() if they expect the Process to live on.
>> > >> >
>> > >> > Policy would not help here actually, javadoc for
>> > >> > java.lang.RuntimePermission says:
>> > >> > "The "exitVM.*" permission is automatically granted to all code 
>> loaded
>> > >> > from the application class path, thus enabling applications to
>> > >> > terminate themselves."
>> > >> > And normally it is hardly possible to bypass the default system
>> > >> > classloader, one need to hack standard classloading delegation
>> > >> > mechanism.
>> > >>
>> > >> Is there any way to influence default policy?
>> > > AFAIU this not a policy, just hardcoded static permissions granted by
>> > > the system classloader when a class is defined - much like any
>> > > URLClassLoader grants access to the class own URL.
>> >
>> > Do you mean that a java applet may bring down a browser process? I had
>> > an impression that applets have a very strict set of permissions, like
>> > no access to the disk, etc. Do they have exit still allowed?
>>
>> Nope until they are loaded with application loader. Normally applet
>> should be loaded by dedicated classloader which does not grant them
>> much permissions + strict (aka sandbox) policy should be active.
>>
>> >
>> > --
>> > Gregory
>> >
>> >
>>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Elena Semukhina <el...@gmail.com>]

On 12/4/06, Evgueni Brevnov <ev...@gmail.com> wrote:
>
> Hi,
>
> I've implemented safe-shutdown of java threads as it was at the
> beginning of this thread. No significant problems were encountered
> while implementing. Find the patch at
> http://issues.apache.org/jira/browse/HARMONY-2391. Here is the
> details:
>
> 1) Java thread stack unwinding in shutdown mode implemented.
> 2) Proper processing of SIGINT (Ctrl+C) implemented.
> 3) Proper processing of SIGQUIT (Ctrl+\) implemented.
> 4) JNI function FatalError reimplemented.
> 5) Smoke test infrastructure extented to compile native sources.
> 6) New smoke tests were added to test VM shutdown.
>
> Note: Enjoy pressing Ctrl+Break to see java stack traces :-) (Ctrl+\ on
> Linux)



Great!
I'm looking forward to pressing Ctrl+Break, Ctrl+\ and especially Ctrl+C!
Thank you!

Elena

Thanks
> Evgueni
>
> On 11/22/06, Alexey Varlamov <al...@gmail.com> wrote:
> > 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> > > Alexey Varlamov wrote:
> > > > 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> > > >> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> > > >> > 2006/11/21, Tim Ellison <t....@gmail.com>:
> > > >> > > Evgueni Brevnov wrote:
> > > >> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > >> > > >>
> > > >> > > >> Evgueni Brevnov wrote:
> > > >> > > >> >
> > > >> > > >> > As it was discussed earlier (and I think agreed) the
> > > >> System.exit()
> > > >> > > >> > should terminate entire process the VM is running in right
> > > >> after it
> > > >> > > >> > executed shutdown hooks and finalizes.
> > > >> > > >>
> > > >> > > >> I don't think so.  How could you embed the VM in another
> program?
> > > >> > > >
> > > >> > > > My initial understanding was that System.exit() shouldn't
> terminate
> > > >> > > > the process. But the experiments proved the opposite. AFAIU
> if you
> > > >> > > > want to embed the application then it should not call
> > > >> System.exit().
> > > >> > >
> > > >> > > I was surprised to see the results of you experiment too -- I'd
> have
> > > >> > > failed that Java pop-quiz question!
> > > >> > >
> > > >> > > > I don't see any problems to extend the proposal to the case
> of
> > > >> > > > System.exit() so that the process will not be stopped. But,
> do
> > > >> we want
> > > >> > > > such incompatibility with RI?
> > > >> > >
> > > >> > > No, we can't change that -- if people expect the program to
> exit()
> > > >> it we
> > > >> > > will see hangs everywhere.  As somebody (I forget) wrote at the
> > > >> time, it
> > > >> > > becomes the responsibility of the app to set the security
> policy to
> > > >> > > disallow exit() if they expect the Process to live on.
> > > >> >
> > > >> > Policy would not help here actually, javadoc for
> > > >> > java.lang.RuntimePermission says:
> > > >> > "The "exitVM.*" permission is automatically granted to all code
> loaded
> > > >> > from the application class path, thus enabling applications to
> > > >> > terminate themselves."
> > > >> > And normally it is hardly possible to bypass the default system
> > > >> > classloader, one need to hack standard classloading delegation
> > > >> > mechanism.
> > > >>
> > > >> Is there any way to influence default policy?
> > > > AFAIU this not a policy, just hardcoded static permissions granted
> by
> > > > the system classloader when a class is defined - much like any
> > > > URLClassLoader grants access to the class own URL.
> > >
> > > Do you mean that a java applet may bring down a browser process? I had
> > > an impression that applets have a very strict set of permissions, like
> > > no access to the disk, etc. Do they have exit still allowed?
> >
> > Nope until they are loaded with application loader. Normally applet
> > should be loaded by dedicated classloader which does not grant them
> > much permissions + strict (aka sandbox) policy should be active.
> >
> > >
> > > --
> > > Gregory
> > >
> > >
> >
>



-- 
Thanks,
Elena

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

Hi,

I've implemented safe-shutdown of java threads as it was at the
beginning of this thread. No significant problems were encountered
while implementing. Find the patch at
http://issues.apache.org/jira/browse/HARMONY-2391. Here is the
details:

1) Java thread stack unwinding in shutdown mode implemented.
2) Proper processing of SIGINT (Ctrl+C) implemented.
3) Proper processing of SIGQUIT (Ctrl+\) implemented.
4) JNI function FatalError reimplemented.
5) Smoke test infrastructure extented to compile native sources.
6) New smoke tests were added to test VM shutdown.

Note: Enjoy pressing Ctrl+Break to see java stack traces :-) (Ctrl+\ on Linux)

Thanks
Evgueni

On 11/22/06, Alexey Varlamov <al...@gmail.com> wrote:
> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> > Alexey Varlamov wrote:
> > > 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> > >> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> > >> > 2006/11/21, Tim Ellison <t....@gmail.com>:
> > >> > > Evgueni Brevnov wrote:
> > >> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > >> > > >>
> > >> > > >> Evgueni Brevnov wrote:
> > >> > > >> >
> > >> > > >> > As it was discussed earlier (and I think agreed) the
> > >> System.exit()
> > >> > > >> > should terminate entire process the VM is running in right
> > >> after it
> > >> > > >> > executed shutdown hooks and finalizes.
> > >> > > >>
> > >> > > >> I don't think so.  How could you embed the VM in another program?
> > >> > > >
> > >> > > > My initial understanding was that System.exit() shouldn't terminate
> > >> > > > the process. But the experiments proved the opposite. AFAIU if you
> > >> > > > want to embed the application then it should not call
> > >> System.exit().
> > >> > >
> > >> > > I was surprised to see the results of you experiment too -- I'd have
> > >> > > failed that Java pop-quiz question!
> > >> > >
> > >> > > > I don't see any problems to extend the proposal to the case of
> > >> > > > System.exit() so that the process will not be stopped. But, do
> > >> we want
> > >> > > > such incompatibility with RI?
> > >> > >
> > >> > > No, we can't change that -- if people expect the program to exit()
> > >> it we
> > >> > > will see hangs everywhere.  As somebody (I forget) wrote at the
> > >> time, it
> > >> > > becomes the responsibility of the app to set the security policy to
> > >> > > disallow exit() if they expect the Process to live on.
> > >> >
> > >> > Policy would not help here actually, javadoc for
> > >> > java.lang.RuntimePermission says:
> > >> > "The "exitVM.*" permission is automatically granted to all code loaded
> > >> > from the application class path, thus enabling applications to
> > >> > terminate themselves."
> > >> > And normally it is hardly possible to bypass the default system
> > >> > classloader, one need to hack standard classloading delegation
> > >> > mechanism.
> > >>
> > >> Is there any way to influence default policy?
> > > AFAIU this not a policy, just hardcoded static permissions granted by
> > > the system classloader when a class is defined - much like any
> > > URLClassLoader grants access to the class own URL.
> >
> > Do you mean that a java applet may bring down a browser process? I had
> > an impression that applets have a very strict set of permissions, like
> > no access to the disk, etc. Do they have exit still allowed?
>
> Nope until they are loaded with application loader. Normally applet
> should be loaded by dedicated classloader which does not grant them
> much permissions + strict (aka sandbox) policy should be active.
>
> >
> > --
> > Gregory
> >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Alexey Varlamov <al...@gmail.com>]

2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> Alexey Varlamov wrote:
> > 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> >> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> >> > 2006/11/21, Tim Ellison <t....@gmail.com>:
> >> > > Evgueni Brevnov wrote:
> >> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >> > > >>
> >> > > >> Evgueni Brevnov wrote:
> >> > > >> >
> >> > > >> > As it was discussed earlier (and I think agreed) the
> >> System.exit()
> >> > > >> > should terminate entire process the VM is running in right
> >> after it
> >> > > >> > executed shutdown hooks and finalizes.
> >> > > >>
> >> > > >> I don't think so.  How could you embed the VM in another program?
> >> > > >
> >> > > > My initial understanding was that System.exit() shouldn't terminate
> >> > > > the process. But the experiments proved the opposite. AFAIU if you
> >> > > > want to embed the application then it should not call
> >> System.exit().
> >> > >
> >> > > I was surprised to see the results of you experiment too -- I'd have
> >> > > failed that Java pop-quiz question!
> >> > >
> >> > > > I don't see any problems to extend the proposal to the case of
> >> > > > System.exit() so that the process will not be stopped. But, do
> >> we want
> >> > > > such incompatibility with RI?
> >> > >
> >> > > No, we can't change that -- if people expect the program to exit()
> >> it we
> >> > > will see hangs everywhere.  As somebody (I forget) wrote at the
> >> time, it
> >> > > becomes the responsibility of the app to set the security policy to
> >> > > disallow exit() if they expect the Process to live on.
> >> >
> >> > Policy would not help here actually, javadoc for
> >> > java.lang.RuntimePermission says:
> >> > "The "exitVM.*" permission is automatically granted to all code loaded
> >> > from the application class path, thus enabling applications to
> >> > terminate themselves."
> >> > And normally it is hardly possible to bypass the default system
> >> > classloader, one need to hack standard classloading delegation
> >> > mechanism.
> >>
> >> Is there any way to influence default policy?
> > AFAIU this not a policy, just hardcoded static permissions granted by
> > the system classloader when a class is defined - much like any
> > URLClassLoader grants access to the class own URL.
>
> Do you mean that a java applet may bring down a browser process? I had
> an impression that applets have a very strict set of permissions, like
> no access to the disk, etc. Do they have exit still allowed?

Nope until they are loaded with application loader. Normally applet
should be loaded by dedicated classloader which does not grant them
much permissions + strict (aka sandbox) policy should be active.

>
> --
> Gregory
>
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Gregory Shimansky <gs...@gmail.com>]

Alexey Varlamov wrote:
> 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
>> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
>> > 2006/11/21, Tim Ellison <t....@gmail.com>:
>> > > Evgueni Brevnov wrote:
>> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>> > > >>
>> > > >> Evgueni Brevnov wrote:
>> > > >> >
>> > > >> > As it was discussed earlier (and I think agreed) the 
>> System.exit()
>> > > >> > should terminate entire process the VM is running in right 
>> after it
>> > > >> > executed shutdown hooks and finalizes.
>> > > >>
>> > > >> I don't think so.  How could you embed the VM in another program?
>> > > >
>> > > > My initial understanding was that System.exit() shouldn't terminate
>> > > > the process. But the experiments proved the opposite. AFAIU if you
>> > > > want to embed the application then it should not call 
>> System.exit().
>> > >
>> > > I was surprised to see the results of you experiment too -- I'd have
>> > > failed that Java pop-quiz question!
>> > >
>> > > > I don't see any problems to extend the proposal to the case of
>> > > > System.exit() so that the process will not be stopped. But, do 
>> we want
>> > > > such incompatibility with RI?
>> > >
>> > > No, we can't change that -- if people expect the program to exit() 
>> it we
>> > > will see hangs everywhere.  As somebody (I forget) wrote at the 
>> time, it
>> > > becomes the responsibility of the app to set the security policy to
>> > > disallow exit() if they expect the Process to live on.
>> >
>> > Policy would not help here actually, javadoc for
>> > java.lang.RuntimePermission says:
>> > "The "exitVM.*" permission is automatically granted to all code loaded
>> > from the application class path, thus enabling applications to
>> > terminate themselves."
>> > And normally it is hardly possible to bypass the default system
>> > classloader, one need to hack standard classloading delegation
>> > mechanism.
>>
>> Is there any way to influence default policy?
> AFAIU this not a policy, just hardcoded static permissions granted by
> the system classloader when a class is defined - much like any
> URLClassLoader grants access to the class own URL.

Do you mean that a java applet may bring down a browser process? I had 
an impression that applets have a very strict set of permissions, like 
no access to the disk, etc. Do they have exit still allowed?

-- 
Gregory

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> 2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> > On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> > > 2006/11/21, Tim Ellison <t....@gmail.com>:
> > > > Evgueni Brevnov wrote:
> > > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > >>
> > > > >> Evgueni Brevnov wrote:
> > > > >> >
> > > > >> > As it was discussed earlier (and I think agreed) the System.exit()
> > > > >> > should terminate entire process the VM is running in right after it
> > > > >> > executed shutdown hooks and finalizes.
> > > > >>
> > > > >> I don't think so.  How could you embed the VM in another program?
> > > > >
> > > > > My initial understanding was that System.exit() shouldn't terminate
> > > > > the process. But the experiments proved the opposite. AFAIU if you
> > > > > want to embed the application then it should not call System.exit().
> > > >
> > > > I was surprised to see the results of you experiment too -- I'd have
> > > > failed that Java pop-quiz question!
> > > >
> > > > > I don't see any problems to extend the proposal to the case of
> > > > > System.exit() so that the process will not be stopped. But, do we want
> > > > > such incompatibility with RI?
> > > >
> > > > No, we can't change that -- if people expect the program to exit() it we
> > > > will see hangs everywhere.  As somebody (I forget) wrote at the time, it
> > > > becomes the responsibility of the app to set the security policy to
> > > > disallow exit() if they expect the Process to live on.
> > >
> > > Policy would not help here actually, javadoc for
> > > java.lang.RuntimePermission says:
> > > "The "exitVM.*" permission is automatically granted to all code loaded
> > > from the application class path, thus enabling applications to
> > > terminate themselves."
> > > And normally it is hardly possible to bypass the default system
> > > classloader, one need to hack standard classloading delegation
> > > mechanism.
> >
> > Is there any way to influence default policy?
> AFAIU this not a policy, just hardcoded static permissions granted by
> the system classloader when a class is defined - much like any
> URLClassLoader grants access to the class own URL.

Aha, you are right.... I recall the game with an exitVM permission...

> >
> > >
> > > >
> > > > Regards,
> > > > Tim
> > > >
> > > > --
> > > >
> > > > Tim Ellison (t.p.ellison@gmail.com)
> > > > IBM Java technology centre, UK.
> > > >
> > >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Alexey Varlamov <al...@gmail.com>]

2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> > 2006/11/21, Tim Ellison <t....@gmail.com>:
> > > Evgueni Brevnov wrote:
> > > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > >>
> > > >> Evgueni Brevnov wrote:
> > > >> >
> > > >> > As it was discussed earlier (and I think agreed) the System.exit()
> > > >> > should terminate entire process the VM is running in right after it
> > > >> > executed shutdown hooks and finalizes.
> > > >>
> > > >> I don't think so.  How could you embed the VM in another program?
> > > >
> > > > My initial understanding was that System.exit() shouldn't terminate
> > > > the process. But the experiments proved the opposite. AFAIU if you
> > > > want to embed the application then it should not call System.exit().
> > >
> > > I was surprised to see the results of you experiment too -- I'd have
> > > failed that Java pop-quiz question!
> > >
> > > > I don't see any problems to extend the proposal to the case of
> > > > System.exit() so that the process will not be stopped. But, do we want
> > > > such incompatibility with RI?
> > >
> > > No, we can't change that -- if people expect the program to exit() it we
> > > will see hangs everywhere.  As somebody (I forget) wrote at the time, it
> > > becomes the responsibility of the app to set the security policy to
> > > disallow exit() if they expect the Process to live on.
> >
> > Policy would not help here actually, javadoc for
> > java.lang.RuntimePermission says:
> > "The "exitVM.*" permission is automatically granted to all code loaded
> > from the application class path, thus enabling applications to
> > terminate themselves."
> > And normally it is hardly possible to bypass the default system
> > classloader, one need to hack standard classloading delegation
> > mechanism.
>
> Is there any way to influence default policy?
AFAIU this not a policy, just hardcoded static permissions granted by
the system classloader when a class is defined - much like any
URLClassLoader grants access to the class own URL.
>
> >
> > >
> > > Regards,
> > > Tim
> > >
> > > --
> > >
> > > Tim Ellison (t.p.ellison@gmail.com)
> > > IBM Java technology centre, UK.
> > >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> 2006/11/21, Tim Ellison <t....@gmail.com>:
> > Evgueni Brevnov wrote:
> > > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > >>
> > >> Evgueni Brevnov wrote:
> > >> >
> > >> > As it was discussed earlier (and I think agreed) the System.exit()
> > >> > should terminate entire process the VM is running in right after it
> > >> > executed shutdown hooks and finalizes.
> > >>
> > >> I don't think so.  How could you embed the VM in another program?
> > >
> > > My initial understanding was that System.exit() shouldn't terminate
> > > the process. But the experiments proved the opposite. AFAIU if you
> > > want to embed the application then it should not call System.exit().
> >
> > I was surprised to see the results of you experiment too -- I'd have
> > failed that Java pop-quiz question!
> >
> > > I don't see any problems to extend the proposal to the case of
> > > System.exit() so that the process will not be stopped. But, do we want
> > > such incompatibility with RI?
> >
> > No, we can't change that -- if people expect the program to exit() it we
> > will see hangs everywhere.  As somebody (I forget) wrote at the time, it
> > becomes the responsibility of the app to set the security policy to
> > disallow exit() if they expect the Process to live on.
>
> Policy would not help here actually, javadoc for
> java.lang.RuntimePermission says:
> "The "exitVM.*" permission is automatically granted to all code loaded
> from the application class path, thus enabling applications to
> terminate themselves."
> And normally it is hardly possible to bypass the default system
> classloader, one need to hack standard classloading delegation
> mechanism.

Is there any way to influence default policy?

>
> >
> > Regards,
> > Tim
> >
> > --
> >
> > Tim Ellison (t.p.ellison@gmail.com)
> > IBM Java technology centre, UK.
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Alexey Varlamov <al...@gmail.com>]

2006/11/21, Tim Ellison <t....@gmail.com>:
> Evgueni Brevnov wrote:
> > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >>
> >> Evgueni Brevnov wrote:
> >> >
> >> > As it was discussed earlier (and I think agreed) the System.exit()
> >> > should terminate entire process the VM is running in right after it
> >> > executed shutdown hooks and finalizes.
> >>
> >> I don't think so.  How could you embed the VM in another program?
> >
> > My initial understanding was that System.exit() shouldn't terminate
> > the process. But the experiments proved the opposite. AFAIU if you
> > want to embed the application then it should not call System.exit().
>
> I was surprised to see the results of you experiment too -- I'd have
> failed that Java pop-quiz question!
>
> > I don't see any problems to extend the proposal to the case of
> > System.exit() so that the process will not be stopped. But, do we want
> > such incompatibility with RI?
>
> No, we can't change that -- if people expect the program to exit() it we
> will see hangs everywhere.  As somebody (I forget) wrote at the time, it
> becomes the responsibility of the app to set the security policy to
> disallow exit() if they expect the Process to live on.

Policy would not help here actually, javadoc for
java.lang.RuntimePermission says:
"The "exitVM.*" permission is automatically granted to all code loaded
from the application class path, thus enabling applications to
terminate themselves."
And normally it is hardly possible to bypass the default system
classloader, one need to hack standard classloading delegation
mechanism.

>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Tim Ellison <t....@gmail.com> wrote:
> Evgueni Brevnov wrote:
> > On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >>
> >> Evgueni Brevnov wrote:
> >> >
> >> > As it was discussed earlier (and I think agreed) the System.exit()
> >> > should terminate entire process the VM is running in right after it
> >> > executed shutdown hooks and finalizes.
> >>
> >> I don't think so.  How could you embed the VM in another program?
> >
> > My initial understanding was that System.exit() shouldn't terminate
> > the process. But the experiments proved the opposite. AFAIU if you
> > want to embed the application then it should not call System.exit().
>
> I was surprised to see the results of you experiment too -- I'd have
> failed that Java pop-quiz question!
>
> > I don't see any problems to extend the proposal to the case of
> > System.exit() so that the process will not be stopped. But, do we want
> > such incompatibility with RI?
>
> No, we can't change that -- if people expect the program to exit() it we
> will see hangs everywhere.  As somebody (I forget) wrote at the time, it
> becomes the responsibility of the app to set the security policy to
> disallow exit() if they expect the Process to live on.

It was Gregory Shimansky :-)

>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Evgueni Brevnov wrote:
> On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>> Evgueni Brevnov wrote:
>> >
>> > As it was discussed earlier (and I think agreed) the System.exit()
>> > should terminate entire process the VM is running in right after it
>> > executed shutdown hooks and finalizes.
>>
>> I don't think so.  How could you embed the VM in another program?
> 
> My initial understanding was that System.exit() shouldn't terminate
> the process. But the experiments proved the opposite. AFAIU if you
> want to embed the application then it should not call System.exit().

I was surprised to see the results of you experiment too -- I'd have
failed that Java pop-quiz question!

> I don't see any problems to extend the proposal to the case of
> System.exit() so that the process will not be stopped. But, do we want
> such incompatibility with RI?

No, we can't change that -- if people expect the program to exit() it we
will see hangs everywhere.  As somebody (I forget) wrote at the time, it
becomes the responsibility of the app to set the security policy to
disallow exit() if they expect the Process to live on.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Tim Ellison wrote:
>>> Geir Magnusson Jr. wrote:
>>>> So VMs aren't embeddable.
>>> How so?  *Any* thread in an app can potentially call an OS's exit() and
>>> bring down the app, so why are you distinguishing the thread that
>>> happens to be running Java code?
>> Becuase I'm reading the spec which says that System.exit() exits the VM,
>> not the process, but the behavior is that current impls kill the process.
> 
> I was responding to your "embeddable" comment.  I agree that the spec
> says that System.exit() will terminate the VM; it does not say that it
> will terminate the process.  However, it's a pretty big compatibility
> issue so I suggest we do likewise.
> 
>>>> Maybe we can offer a flag for people that want to truly embed the VM...
>>> If you don't want the Java app to call System.exit() then you can set up
>>> a security manager to prevent it.  It will then be required to unwind
>>> nicely to get out, and the VM will terminate when there are no
>>> non-daemon threads etc etc.
>> But there's nothing wrong with System.exit() just killing the VM.  So
>> sure, you can revoke the exitVM perm, but also in the event when you
>> want it to happen, I think there's nothing wrong with letting it, and
>> just having the VM exit.
> 
> No, we have to be compatible here.  There may be apps that use libraries
> that fork worker threads that are nothing to do with Java -- they will
> 'hang' (i.e. continue running) if we don't behave like other
> implementations.

We will be compatible.  But there's no harm in offering a configuration 
option that lets people who embed make a *conscious* decision to use it 
'correctly'.

geir

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> Tim Ellison wrote:
>> Geir Magnusson Jr. wrote:
>>> So VMs aren't embeddable.
>>
>> How so?  *Any* thread in an app can potentially call an OS's exit() and
>> bring down the app, so why are you distinguishing the thread that
>> happens to be running Java code?
> 
> Becuase I'm reading the spec which says that System.exit() exits the VM,
> not the process, but the behavior is that current impls kill the process.

I was responding to your "embeddable" comment.  I agree that the spec
says that System.exit() will terminate the VM; it does not say that it
will terminate the process.  However, it's a pretty big compatibility
issue so I suggest we do likewise.

>>> Maybe we can offer a flag for people that want to truly embed the VM...
>>
>> If you don't want the Java app to call System.exit() then you can set up
>> a security manager to prevent it.  It will then be required to unwind
>> nicely to get out, and the VM will terminate when there are no
>> non-daemon threads etc etc.
> 
> But there's nothing wrong with System.exit() just killing the VM.  So
> sure, you can revoke the exitVM perm, but also in the event when you
> want it to happen, I think there's nothing wrong with letting it, and
> just having the VM exit.

No, we have to be compatible here.  There may be apps that use libraries
that fork worker threads that are nothing to do with Java -- they will
'hang' (i.e. continue running) if we don't behave like other
implementations.

I expect you will get better mileage in lobbying to get the spec changed
to reflect reality.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> So VMs aren't embeddable.
> 
> How so?  *Any* thread in an app can potentially call an OS's exit() and
> bring down the app, so why are you distinguishing the thread that
> happens to be running Java code?

Becuase I'm reading the spec which says that System.exit() exits the VM, 
not the process, but the behavior is that current impls kill the process.

> 
>> <sigh>
>>
>> Maybe we can offer a flag for people that want to truly embed the VM...
> 
> If you don't want the Java app to call System.exit() then you can set up
> a security manager to prevent it.  It will then be required to unwind
> nicely to get out, and the VM will terminate when there are no
> non-daemon threads etc etc.


But there's nothing wrong with System.exit() just killing the VM.  So 
sure, you can revoke the exitVM perm, but also in the event when you 
want it to happen, I think there's nothing wrong with letting it, and 
just having the VM exit.

geir

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> So VMs aren't embeddable.

How so?  *Any* thread in an app can potentially call an OS's exit() and
bring down the app, so why are you distinguishing the thread that
happens to be running Java code?

> <sigh>
> 
> Maybe we can offer a flag for people that want to truly embed the VM...

If you don't want the Java app to call System.exit() then you can set up
a security manager to prevent it.  It will then be required to unwind
nicely to get out, and the VM will terminate when there are no
non-daemon threads etc etc.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Evgueni Brevnov wrote:
>>> On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>>>
>>>> Evgueni Brevnov wrote:
>>>>> As it was discussed earlier (and I think agreed) the System.exit()
>>>>> should terminate entire process the VM is running in right after it
>>>>> executed shutdown hooks and finalizes.
>>>> I don't think so.  How could you embed the VM in another program?
>>>  My initial understanding was that System.exit() shouldn't terminate
>>> the process. But the experiments proved the opposite. AFAIU if you
>>> want to embed the application then it should not call System.exit().
>>>  I don't see any problems to extend the proposal to the case of
>>> System.exit() so that the process will not be stopped. But, do we want
>>> such incompatibility with RI?
>> Because the RI is doing something really stupid?
>>
>> It's supposed to just terminate the VM, right?
>>
>> IBM-ers... what does J9 do, and why?
> 
> System.exit() eventually results in a call to exit() (on most OSes), so
> it will bring down the process too.
> 
> I don't think Harmony can change that because it may well break apps.

So VMs aren't embeddable.

<sigh>

Maybe we can offer a flag for people that want to truly embed the VM...

geir

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Evgueni Brevnov wrote:
> Tim, I agree with you here. We are just trying to figure out how an
> app can protect itself to be protected against System.exit()

Just revoke the RuntimePermission("exitVM").

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Tim Ellison <t....@gmail.com> wrote:
> Geir Magnusson Jr. wrote:
> > Evgueni Brevnov wrote:
> >> On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >>>
> >>>
> >>> Evgueni Brevnov wrote:
> >>> >
> >>> > As it was discussed earlier (and I think agreed) the System.exit()
> >>> > should terminate entire process the VM is running in right after it
> >>> > executed shutdown hooks and finalizes.
> >>>
> >>> I don't think so.  How could you embed the VM in another program?
> >>
> >>  My initial understanding was that System.exit() shouldn't terminate
> >> the process. But the experiments proved the opposite. AFAIU if you
> >> want to embed the application then it should not call System.exit().
> >>  I don't see any problems to extend the proposal to the case of
> >> System.exit() so that the process will not be stopped. But, do we want
> >> such incompatibility with RI?
> >
> > Because the RI is doing something really stupid?
> >
> > It's supposed to just terminate the VM, right?
> >
> > IBM-ers... what does J9 do, and why?
>
> System.exit() eventually results in a call to exit() (on most OSes), so
> it will bring down the process too.
>
> I don't think Harmony can change that because it may well break apps.

Tim, I agree with you here. We are just trying to figure out how an
app can protect itself to be protected against System.exit()

>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> Evgueni Brevnov wrote:
>> On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>>
>>>
>>> Evgueni Brevnov wrote:
>>> >
>>> > As it was discussed earlier (and I think agreed) the System.exit()
>>> > should terminate entire process the VM is running in right after it
>>> > executed shutdown hooks and finalizes.
>>>
>>> I don't think so.  How could you embed the VM in another program?
>>
>>  My initial understanding was that System.exit() shouldn't terminate
>> the process. But the experiments proved the opposite. AFAIU if you
>> want to embed the application then it should not call System.exit().
>>  I don't see any problems to extend the proposal to the case of
>> System.exit() so that the process will not be stopped. But, do we want
>> such incompatibility with RI?
> 
> Because the RI is doing something really stupid?
> 
> It's supposed to just terminate the VM, right?
> 
> IBM-ers... what does J9 do, and why?

System.exit() eventually results in a call to exit() (on most OSes), so
it will bring down the process too.

I don't think Harmony can change that because it may well break apps.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

Evgueni Brevnov wrote:
> On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>>
>> Evgueni Brevnov wrote:
>> >
>> > As it was discussed earlier (and I think agreed) the System.exit()
>> > should terminate entire process the VM is running in right after it
>> > executed shutdown hooks and finalizes.
>>
>> I don't think so.  How could you embed the VM in another program?
> 
>  My initial understanding was that System.exit() shouldn't terminate
> the process. But the experiments proved the opposite. AFAIU if you
> want to embed the application then it should not call System.exit().
>  I don't see any problems to extend the proposal to the case of
> System.exit() so that the process will not be stopped. But, do we want
> such incompatibility with RI?

Because the RI is doing something really stupid?

It's supposed to just terminate the VM, right?

IBM-ers... what does J9 do, and why?

geir

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
> Evgueni Brevnov wrote:
> >
> > As it was discussed earlier (and I think agreed) the System.exit()
> > should terminate entire process the VM is running in right after it
> > executed shutdown hooks and finalizes.
>
> I don't think so.  How could you embed the VM in another program?

  My initial understanding was that System.exit() shouldn't terminate
the process. But the experiments proved the opposite. AFAIU if you
want to embed the application then it should not call System.exit().
  I don't see any problems to extend the proposal to the case of
System.exit() so that the process will not be stopped. But, do we want
such incompatibility with RI?

Evgueni
>
> geir
>

Re: [drlvm][shutdown] How to cleanup resources safely?

["Geir Magnusson Jr." <ge...@pobox.com>]

Evgueni Brevnov wrote:
> 
> As it was discussed earlier (and I think agreed) the System.exit()
> should terminate entire process the VM is running in right after it
> executed shutdown hooks and finalizes. 

I don't think so.  How could you embed the VM in another program?

geir

Re: [drlvm][shutdown] How to cleanup resources safely?

[Tim Ellison <t....@gmail.com>]

Thanks for discussing this, it's important...

Evgueni Brevnov wrote:
> Hi All,
> 
> I'd like to share with you my findings regarding current DRLVM
> shutdown procedure (not in too many details) and discuss further steps
> to improve it.
> 
> /* General words about shutdown */
> 
> Here is a snip from the j.l.Runtime.addShutdownHook() spec:
> 
> "The Java virtual machine shuts down in response to two kinds of events:
> 1) The program exits normally, when the last non-daemon thread exits
> or when the exit (equivalently, System.exit) method is invoked, or
> 2) The virtual machine is terminated in response to a user interrupt,
> such as typing ^C, or a system-wide event, such as user logoff or
> system shutdown."
> 
> Except these 2 events there is one more player on the scene. It is
> DestroyJavaVM defined by invocation API.User calls DestroyJavaVM to
> request reclamation of  all resources owned by the VM.
> 
> As it was discussed earlier (and I think agreed) the System.exit()
> should terminate entire process the VM is running in right after it
> executed shutdown hooks and finalizes. At least RI does so. Thus I
> would like to look closer at the case when the last non-daemon thread
> exits and the VM is requested to cleanup resources through
> DestroyJavaVM. When DestroyJavaVM is called it waits until current
> thread is the last non-daemon thread before cleaning up all resources
> owned by the VM. It is important to note it is still possible daemon
> java and native threads are running inside the VM even if there is no
> running non-daemon threads.  These running threads may still be using
> VM-wide data (generally located in Global_Env). Thus before cleaning
> resources up we need to stop these threads somehow. Unfortunately, we
> can't just kill threads. We need to release all resources (system
> locks, heap/stack memory, open files/connections, etc) owed by each
> thread. So the problem I'd like to discuss and solve: How to stop
> running threads in a safe manner?
> 
> /* Current DRLVM implementation */
> 
> Currently, the DRLVM isn't able to stop running threads in a safe way.
> Thus it just doesn't clean resources up if there are running threads
> after all non-daemon threads exits.
> 
> /* How to stop threads safely. Proposal. */
> 
> I propose to consider native and java threads separately. Lets start
> with native threads.
> 
> Native threads: It seems there is not much we can do to stop arbitrary
> native thread and release resources owed by it. I don't consider
> global resource management as a possible solution here :-). Thus there
> are two requirements for native threads to ensure safe VM shutdown:
> 
> a) Any native thread must be created by TM or attached to TM. So TM is
> aware about all running native threads.

I don't believe you can mandate 'must' here, it is unenforceable.

> b) Native thread provides a callback which is called to request thread
> shutdown. It's the native thread responsibility to clean its resources
> upon shutdown request.

Agreed, that is a sensible solution.  Any threads that know they need
clean-up work on shutdown may register to be informed when the shutdown
is about to occur.  I expect we will want describe the state of the VM
at that point, and put restrictions on what threads can do in that
callback function.

> Java threads: We have much more control other java threads. Namely it
> is possible to unwind the stack in a safe way. When the VM needs to
> stop a java thread it asynchronously raises an exception to that
> thread. The VM ensures that the exception will not be caught by any
> java exception handler (ala j.l.ThreadDeath exception). This guarantee
> full and safe java stack unwinding. If there is no user native frames
> on the stack (JNI calls) then thread exits normally otherwise the
> control is returned to the user's code with an exception raised. It is
> the user's responsibility to handle exception properly and clean up
> all resources allocated by native code.

Sounds good.

> Any suggestions/comments are HIGHLY welcome.

I think this can be quite a generic application resource lifecycle API,
and the API should be put into a library that the classlib natives can
share.  In that case the classlib code can register clean-up functions
for open system resources too.  Sounds like a VMI extension to me.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [drlvm][shutdown] How to cleanup resources safely?

[Alexey Varlamov <al...@gmail.com>]

2006/11/21, Evgueni Brevnov <ev...@gmail.com>:
> On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> > [snip]
> > > Java threads: We have much more control other java threads. Namely it
> > > is possible to unwind the stack in a safe way. When the VM needs to
> > > stop a java thread it asynchronously raises an exception to that
> > > thread. The VM ensures that the exception will not be caught by any
> > > java exception handler (ala j.l.ThreadDeath exception). This guarantee
> > > full and safe java stack unwinding. If there is no user native frames
> > > on the stack (JNI calls) then thread exits normally otherwise the
> > > control is returned to the user's code with an exception raised. It is
> > > the user's responsibility to handle exception properly and clean up
> > > all resources allocated by native code.
> > This part looks a bit contradictory: on the one hand, "VM ensures the
> > exception will not be caugth", and on the other "control is returned
> > to user's code with the exception raised". I presume you made a slip
> > on the last and actually meant we can unwind safely and free
> > stack-related VM resources; but then we should postpone shutdown
> > hooks/finalizers until no other Java threads running including
> > daemons?
>
> Alexey, I really appreciate your interest to this topic.
> Unfortunately, I can't fully understand what you are asking about..
> :-( The idea is we are able to unwind java and VM native stack with
> cleaning resources on the way. Unfortunately we can't do the same with
> user native frames. So we propagate an exception up to the most recent
> user frame if any. The user code should be ready to process the
> exception and release allocated resources if any. All this is
> consistent with the JNI spec. Here is the snip from the spec regarding
> asynchronous exceptions.
>
> <SNIP>
> Asynchronous Exceptions
> In cases of multiple threads, threads other than the current thread
> may post an asynchronous exception. An asynchronous exception does not
> immediately affect the execution of the native code in the current
> thread, until:
>
> the native code calls one of the JNI functions that could raise
> synchronous exceptions, or
> the native code uses ExceptionOccurred() to explicitly check for
> synchronous and asynchronous exceptions.
> Note that only those JNI function that could potentially raise
> synchronous exceptions check for asynchronous exceptions.
>
> Native methods should insert ExceptionOccurred()checks in necessary
> places (such as in a tight loop without other exception checks) to
> ensure that the current thread responds to asynchronous exceptions in
> a reasonable amount of time.
> </SNIP>
Ah, it's me who misunderstood the initial wording; now this is quite clear.
Thanks!

>
> > Other paragraphs look super, well-though. Great job!
> Glad to hear that.
>
> Thanks
> Evgueni
> >
> > >
> > > Any suggestions/comments are HIGHLY welcome.
> > >
> > > Thanks
> > > Evgueni
> > >
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Evgueni Brevnov <ev...@gmail.com>]

On 11/21/06, Alexey Varlamov <al...@gmail.com> wrote:
> [snip]
> > Java threads: We have much more control other java threads. Namely it
> > is possible to unwind the stack in a safe way. When the VM needs to
> > stop a java thread it asynchronously raises an exception to that
> > thread. The VM ensures that the exception will not be caught by any
> > java exception handler (ala j.l.ThreadDeath exception). This guarantee
> > full and safe java stack unwinding. If there is no user native frames
> > on the stack (JNI calls) then thread exits normally otherwise the
> > control is returned to the user's code with an exception raised. It is
> > the user's responsibility to handle exception properly and clean up
> > all resources allocated by native code.
> This part looks a bit contradictory: on the one hand, "VM ensures the
> exception will not be caugth", and on the other "control is returned
> to user's code with the exception raised". I presume you made a slip
> on the last and actually meant we can unwind safely and free
> stack-related VM resources; but then we should postpone shutdown
> hooks/finalizers until no other Java threads running including
> daemons?

Alexey, I really appreciate your interest to this topic.
Unfortunately, I can't fully understand what you are asking about..
:-( The idea is we are able to unwind java and VM native stack with
cleaning resources on the way. Unfortunately we can't do the same with
user native frames. So we propagate an exception up to the most recent
user frame if any. The user code should be ready to process the
exception and release allocated resources if any. All this is
consistent with the JNI spec. Here is the snip from the spec regarding
asynchronous exceptions.

<SNIP>
Asynchronous Exceptions
In cases of multiple threads, threads other than the current thread
may post an asynchronous exception. An asynchronous exception does not
immediately affect the execution of the native code in the current
thread, until:

the native code calls one of the JNI functions that could raise
synchronous exceptions, or
the native code uses ExceptionOccurred() to explicitly check for
synchronous and asynchronous exceptions.
Note that only those JNI function that could potentially raise
synchronous exceptions check for asynchronous exceptions.

Native methods should insert ExceptionOccurred()checks in necessary
places (such as in a tight loop without other exception checks) to
ensure that the current thread responds to asynchronous exceptions in
a reasonable amount of time.
</SNIP>


> Other paragraphs look super, well-though. Great job!
Glad to hear that.

Thanks
Evgueni
>
> >
> > Any suggestions/comments are HIGHLY welcome.
> >
> > Thanks
> > Evgueni
> >
>

Re: [drlvm][shutdown] How to cleanup resources safely?

[Alexey Varlamov <al...@gmail.com>]

[snip]
> Java threads: We have much more control other java threads. Namely it
> is possible to unwind the stack in a safe way. When the VM needs to
> stop a java thread it asynchronously raises an exception to that
> thread. The VM ensures that the exception will not be caught by any
> java exception handler (ala j.l.ThreadDeath exception). This guarantee
> full and safe java stack unwinding. If there is no user native frames
> on the stack (JNI calls) then thread exits normally otherwise the
> control is returned to the user's code with an exception raised. It is
> the user's responsibility to handle exception properly and clean up
> all resources allocated by native code.
This part looks a bit contradictory: on the one hand, "VM ensures the
exception will not be caugth", and on the other "control is returned
to user's code with the exception raised". I presume you made a slip
on the last and actually meant we can unwind safely and free
stack-related VM resources; but then we should postpone shutdown
hooks/finalizers until no other Java threads running including
daemons?
Other paragraphs look super, well-though. Great job!

>
> Any suggestions/comments are HIGHLY welcome.
>
> Thanks
> Evgueni
>