Forked large unit tests seem slow to terminate

[Andrew Purtell <ap...@apache.org>]

We eventually have no zombies, at least the zombie detector does not find
anything, but meanwhile on memory limited EC2 instances I'm seeing tests
killed by the OOM killer, very likely meaning that previous tests are still
in the process of shutting down and are hanging around in the process
table, consuming RAM, VMM, etc. I didn't see this before the switch to
Hadoop 2 as default. I'm not suggesting we switch back. I wonder if there
is anything that can be done to wait for the forked process to actually
terminate? Is this a Surefire bug?

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Forked large unit tests seem slow to terminate

[Andrew Purtell <ap...@apache.org>]

Hmm... Let me try a shutdown hook for the minicluster that commits hari
kari after 10 seconds.


On Fri, Dec 13, 2013 at 1:15 AM, Nicolas Liochon <nk...@gmail.com> wrote:

> > I wonder if there is anything that can be done to wait for the forked
> process to actually terminate?
> As far as I know, no. We could be hacky, and, in the afterClass, start a
> deamon thread that will do a kill -9 after a few seconds. Or something
> similar, but that's very hacky imho.
>
> > Is this a Surefire bug?
> I don't think so, even if surefire has its share of bugs.
> The issue could be in HBase itself: surefire execute the "afterClass"
> method, this method does finish, but HBase then gets stuck in the shutdown
> hooks.
>
>
>
>
> On Fri, Dec 13, 2013 at 1:26 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > We eventually have no zombies, at least the zombie detector does not find
> > anything, but meanwhile on memory limited EC2 instances I'm seeing tests
> > killed by the OOM killer, very likely meaning that previous tests are
> still
> > in the process of shutting down and are hanging around in the process
> > table, consuming RAM, VMM, etc. I didn't see this before the switch to
> > Hadoop 2 as default. I'm not suggesting we switch back. I wonder if there
> > is anything that can be done to wait for the forked process to actually
> > terminate? Is this a Surefire bug?
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Forked large unit tests seem slow to terminate

[Nicolas Liochon <nk...@gmail.com>]

> I wonder if there is anything that can be done to wait for the forked
process to actually terminate?
As far as I know, no. We could be hacky, and, in the afterClass, start a
deamon thread that will do a kill -9 after a few seconds. Or something
similar, but that's very hacky imho.

> Is this a Surefire bug?
I don't think so, even if surefire has its share of bugs.
The issue could be in HBase itself: surefire execute the "afterClass"
method, this method does finish, but HBase then gets stuck in the shutdown
hooks.




On Fri, Dec 13, 2013 at 1:26 AM, Andrew Purtell <ap...@apache.org> wrote:

> We eventually have no zombies, at least the zombie detector does not find
> anything, but meanwhile on memory limited EC2 instances I'm seeing tests
> killed by the OOM killer, very likely meaning that previous tests are still
> in the process of shutting down and are hanging around in the process
> table, consuming RAM, VMM, etc. I didn't see this before the switch to
> Hadoop 2 as default. I'm not suggesting we switch back. I wonder if there
> is anything that can be done to wait for the forked process to actually
> terminate? Is this a Surefire bug?
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>