You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@river.apache.org by Sim IJskes - QCG <si...@qcg.nl> on 2010/11/26 13:49:47 UTC

QA speed improvement

Suppose we have a good kill mechanism for all the testing slave VMs,
would there be any harm in activating this kill mechanism during 
teardown? So no orderly teardown, but a brute kill.

Gr. Sim

-- 
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397

Re: QA speed improvement

Posted by Gregg Wonderly <gr...@wonderly.org>.

In many cases, there is not a shutdown hook registered when there could be. 
Reggie, Mahalo and Norm in particular, having "subscription" based use, should 
notify their users immediately apon catchable VM shutdown.  The mechanism 
provided by

	Runtime.getRuntime().addShutdownHook(Runnable)

should be used everywhere we can to "speed up" the notification of process failures.

Gregg Wonderly

On 11/26/2010 8:21 AM, Patricia Shanahan wrote:
> Sim IJskes - QCG wrote:
>> On 26-11-10 14:42, Patricia Shanahan wrote:
>>>>> 2. Ensuring that the River features that are used during the current
>>>>> teardown are tested in some other way.
>>>>
>>>> But only if the teardown is already part of the testresult. Is it?
>>>
>>> Whether or not a teardown failure gets reflected in the test result, we
>>> would learn about it soon enough under current conditions, due e.g. to
>>> non-running of subsequent tests. I suspect we do have the formal test
>>> coverage, but I'll take a look at what teardown uses now, and check.
>>
>> What worries me a little is the state the operating system keeps on terminated
>> processes. We know of the SO_REUSEADDR issue, but you seem to experience the
>> non-release of resources on process termination. formally a OS bug, but if so
>> is something we need to workaround. How definate are you on your reports where
>> you have problems rerunning a test caused by the OS not releasing resources on
>> VM exit?
>
> I see two issues here:
>
> 1. Terminating every VM.
>
> 2. Deleting any temporary files.
>
> Empirically, a Ctrl-C termination often leaves the system in a state in which
> subsequent tests fail.
>
> Patricia
>

Re: QA speed improvement

Posted by Gregg Wonderly <gr...@wonderly.org>.

On 11/26/2010 10:32 PM, Patricia Shanahan wrote:
> Patricia Shanahan wrote:
>> Sim IJskes - QCG wrote:
>>> On 26-11-10 15:21, Patricia Shanahan wrote:
>>>>> What worries me a little is the state the operating system keeps on
>>>>> terminated processes. We know of the SO_REUSEADDR issue, but you seem
>>>>> to experience the non-release of resources on process termination.
>>>>> formally a OS bug, but if so is something we need to workaround. How
>>>>> definate are you on your reports where you have problems rerunning a
>>>>> test caused by the OS not releasing resources on VM exit?
>>>>
>>>> I see two issues here:
>>>>
>>>> 1. Terminating every VM.
>>>
>>> A working termination is a firm requirement before bypassing orderly teardown
>>> can be attempted.
>>>
>>>> 2. Deleting any temporary files.
>>
>> I think this comes down to making sure that if we create any files they
>> are marked deleteOnExit, reducing the problem to terminating every VM.
>>
>>>>
>>>> Empirically, a Ctrl-C termination often leaves the system in a state in
>>>> which subsequent tests fail.
>>>
>>> Any idea which VM receives this Ctrl-C? The ant VM, the harness master, or
>>> any random process? There is no such concept as process group leader under
>>> windows is there?
>>
>> I'm sure the shell initially delivers the signal to the ant job that I
>> told it to run. I have no idea how and to what extent it gets passed on
>> to the various processes that are created e.g. to run services. It's a
>> good question, and I'll think a bit about how to find out.
>
> I've thought of two schemes, and some reasons why we may want to implement both.
>
> Scheme 1: Modify each class that calls Runtime.getRuntime.exec to register a
> shutdownHook to destroy the Process that exec returned.
>
> This has a window between the exec call and the addShutdownHook call during
> which termination of the VM doing the process creation would leave the process
> as an orphan, with no arrangement to destroy it.
>
> Scheme 2: Require each public static void main(String[]) method in River to log
> the pid of its process in a log message with a specific format. Write a program
> that scans the log file for those messages, and kills each process.
>
> This is reliable, in the sense that a thread can be required to write to the log
> before doing anything that reserves a resource. It is not so good for normal
> shutdown, because it requires log reading.
>
> I think Scheme 1 would work well for normal termination. By the time the test
> finishes, all the windows between creating a process and creating the shutdown
> hook to kill it will have closed. Scheme 2 would be useful as a backup in the
> event of a crash or cntrl-C while processes are being created. Knowing the PID
> of each River process in a configuration could be useful to admins running
> production systems, as well as in our QA environment.
>
> Any other ideas?

I believe that in UNIX, #1 is already taken care of for interactive sessions 
through the propagation of SIGINT to all child processes by the kernel.  But, as 
I said before, Runtime.addShutdownHook() would be a much more reliable way to 
manage the shutdown process.  Adding one shutdown thread per background task 
would not be advisable.  Instead, we'd need to track the "background threads" in 
a Collection, and have one shutdown hook that inspected that collection and did 
the appropriate thing.

Creating the "test" processes from a single "process" that used such a mechanism 
would allow for a more dependable startup and shutdown.  In that case, one 
thread per background process would be appropriate because we could then use 
parallel threads (2x the number of processes) to Thread.join() the shutdown hook 
threads, and report when they were shutdown etc.

Gregg