You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Hanifi Gunes <hg...@maprtech.com> on 2015/05/14 02:26:50 UTC
forked process starves memory and dies
I started constantly hitting this issue on my linux VM with 8 logical
cores. What appears to be happing is forked process is being killed due to
memory starvation. I see that we are running surefire with max direct
memory reservation of 4gb initially.
On average 3G/8 forks makes 375M per fork and so per test suite. I would
think that this should be good enough for many suites but do we have any
test suites where we make huge allocations? Last but not least this may
indicate we are leaking memory.
*[INFO]
------------------------------------------------------------------------*
*[INFO] BUILD FAILURE*
*[INFO]
------------------------------------------------------------------------*
*[INFO] Total time: 06:04 min*
*[INFO] Finished at: 2015-05-14T00:07:26+00:00*
*[INFO] Final Memory: 18M/380M*
*[INFO]
------------------------------------------------------------------------*
*[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-cli) on
project drill-java-exec: ExecutionException: java.lang.RuntimeException:
The forked VM terminated without properly saying goodbye. VM crash or
System.exit called?*
-Hanifi
Re: forked process starves memory and dies
Posted by Hanifi Gunes <hg...@maprtech.com>.
I was reverse reading the situation considering that Drill was starving and
exiting. It was OS killing the fork. Looks like memory is never been
returned back to OS causing starvation. Reducing max direct memory fixed
the problem.
On Wed, May 13, 2015 at 7:53 PM, Hanifi Gunes <hg...@maprtech.com> wrote:
> I thought total, my bad. Any other idea why the forked process has been
> killed though?
>
> On Wed, May 13, 2015 at 6:33 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
>> The 4gb is per fork, not total.
>>
>> On Wed, May 13, 2015 at 5:26 PM, Hanifi Gunes <hg...@maprtech.com>
>> wrote:
>>
>> > I started constantly hitting this issue on my linux VM with 8 logical
>> > cores. What appears to be happing is forked process is being killed due
>> to
>> > memory starvation. I see that we are running surefire with max direct
>> > memory reservation of 4gb initially.
>> >
>> >
>> > On average 3G/8 forks makes 375M per fork and so per test suite. I would
>> > think that this should be good enough for many suites but do we have any
>> > test suites where we make huge allocations? Last but not least this may
>> > indicate we are leaking memory.
>> >
>> > *[INFO]
>> >
>> ------------------------------------------------------------------------*
>> >
>> > *[INFO] BUILD FAILURE*
>> >
>> > *[INFO]
>> >
>> ------------------------------------------------------------------------*
>> >
>> > *[INFO] Total time: 06:04 min*
>> >
>> > *[INFO] Finished at: 2015-05-14T00:07:26+00:00*
>> >
>> > *[INFO] Final Memory: 18M/380M*
>> >
>> > *[INFO]
>> >
>> ------------------------------------------------------------------------*
>> >
>> > *[ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-cli)
>> on
>> > project drill-java-exec: ExecutionException: java.lang.RuntimeException:
>> > The forked VM terminated without properly saying goodbye. VM crash or
>> > System.exit called?*
>> >
>> >
>> > -Hanifi
>> >
>>
>
>
Re: forked process starves memory and dies
Posted by Hanifi Gunes <hg...@maprtech.com>.
I thought total, my bad. Any other idea why the forked process has been
killed though?
On Wed, May 13, 2015 at 6:33 PM, Jacques Nadeau <ja...@apache.org> wrote:
> The 4gb is per fork, not total.
>
> On Wed, May 13, 2015 at 5:26 PM, Hanifi Gunes <hg...@maprtech.com> wrote:
>
> > I started constantly hitting this issue on my linux VM with 8 logical
> > cores. What appears to be happing is forked process is being killed due
> to
> > memory starvation. I see that we are running surefire with max direct
> > memory reservation of 4gb initially.
> >
> >
> > On average 3G/8 forks makes 375M per fork and so per test suite. I would
> > think that this should be good enough for many suites but do we have any
> > test suites where we make huge allocations? Last but not least this may
> > indicate we are leaking memory.
> >
> > *[INFO]
> > ------------------------------------------------------------------------*
> >
> > *[INFO] BUILD FAILURE*
> >
> > *[INFO]
> > ------------------------------------------------------------------------*
> >
> > *[INFO] Total time: 06:04 min*
> >
> > *[INFO] Finished at: 2015-05-14T00:07:26+00:00*
> >
> > *[INFO] Final Memory: 18M/380M*
> >
> > *[INFO]
> > ------------------------------------------------------------------------*
> >
> > *[ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-cli) on
> > project drill-java-exec: ExecutionException: java.lang.RuntimeException:
> > The forked VM terminated without properly saying goodbye. VM crash or
> > System.exit called?*
> >
> >
> > -Hanifi
> >
>
Re: forked process starves memory and dies
Posted by Jacques Nadeau <ja...@apache.org>.
The 4gb is per fork, not total.
On Wed, May 13, 2015 at 5:26 PM, Hanifi Gunes <hg...@maprtech.com> wrote:
> I started constantly hitting this issue on my linux VM with 8 logical
> cores. What appears to be happing is forked process is being killed due to
> memory starvation. I see that we are running surefire with max direct
> memory reservation of 4gb initially.
>
>
> On average 3G/8 forks makes 375M per fork and so per test suite. I would
> think that this should be good enough for many suites but do we have any
> test suites where we make huge allocations? Last but not least this may
> indicate we are leaking memory.
>
> *[INFO]
> ------------------------------------------------------------------------*
>
> *[INFO] BUILD FAILURE*
>
> *[INFO]
> ------------------------------------------------------------------------*
>
> *[INFO] Total time: 06:04 min*
>
> *[INFO] Finished at: 2015-05-14T00:07:26+00:00*
>
> *[INFO] Final Memory: 18M/380M*
>
> *[INFO]
> ------------------------------------------------------------------------*
>
> *[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-cli) on
> project drill-java-exec: ExecutionException: java.lang.RuntimeException:
> The forked VM terminated without properly saying goodbye. VM crash or
> System.exit called?*
>
>
> -Hanifi
>