You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by Eddie Epstein <ea...@gmail.com> on 2013/11/07 14:32:40 UTC
Re: Can not create new threads / ducc resource exhaustion while
running RawText Example
Indeed, has been seen before. One interesting post that explains the
problem is here:
http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/
I thought we updated the duccumentation to warn about this, but can't find
it now.
Will get that in now.
Eddie
On Wed, Nov 6, 2013 at 5:33 PM, Neal R Lewis <nr...@us.ibm.com> wrote:
> I rebuilt DUCC yesterday, November 5th, 2013. I'm running the
> DuccRawTextSpec.job with the same GutenBurg books, and
> process_thread_count=24
>
> Machine: Fedora 19 x86_64. 32 Cores, 400GB Mem. Using OpenJDK.
>
>
> Shortly after submitting the job, I get an: "OutOfMemoryError: unable to
> create new native thread. "
>
> I can no longer run any commands as ducc (but can as root or a different
> user) in the the shell, I get either:
> "-bash: fork: retry: No child processes" OR
> "-bash: fork: Resource temporarily unavailable"
>
> The ducc user become unusable.
>
> As far as I can tell, the job finished successfully because I have the
> processed output, and there are still 14 java threads that ducc user is
> running.
> The ducc monitory web server seems to be stuck as well, with the last
> refresh showing the State as Running.
>
> Has anyone seen this before?
>
> Thanks!
> Neal
>
>
>
Re: Can not create new threads / ducc resource exhaustion while running
RawText Example
Posted by Neal R Lewis <nr...@us.ibm.com>.
Thanks Eddie,
I upped ulimit -u and now everything runs smoothly.
I'm now tuning with different parameters.
Thanks!
From: Eddie Epstein <ea...@gmail.com>
To: dev@uima.apache.org
Date: 11/07/2013 08:12 AM
Subject: Re: Can not create new threads / ducc resource exhaustion
while running RawText Example
There is a comment in the "Software Prerequisites" section but it is not
clear.
The docs will be changed to something like:
There are two user limits that may need changing for user ducc, preferably
in ducc's login profile. The default nfiles limit (-n) of 1024 will limit
the
number of broker connections. Raise this to 2x the number of user
processes
running on the cluster. In some Linux OS the default max user process
limit
(-u)
has been reduced to 1024. Raise this up to its hard limit.
Eddie
On Thu, Nov 7, 2013 at 8:32 AM, Eddie Epstein <ea...@gmail.com> wrote:
> Indeed, has been seen before. One interesting post that explains the
> problem is here:
>
>
http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/
>
> I thought we updated the duccumentation to warn about this, but can't
find
> it now.
> Will get that in now.
>
> Eddie
>
>
> On Wed, Nov 6, 2013 at 5:33 PM, Neal R Lewis <nr...@us.ibm.com> wrote:
>
>> I rebuilt DUCC yesterday, November 5th, 2013. I'm running the
>> DuccRawTextSpec.job with the same GutenBurg books, and
>> process_thread_count=24
>>
>> Machine: Fedora 19 x86_64. 32 Cores, 400GB Mem. Using OpenJDK.
>>
>>
>> Shortly after submitting the job, I get an: "OutOfMemoryError: unable
to
>> create new native thread. "
>>
>> I can no longer run any commands as ducc (but can as root or a
different
>> user) in the the shell, I get either:
>> "-bash: fork: retry: No child processes" OR
>> "-bash: fork: Resource temporarily unavailable"
>>
>> The ducc user become unusable.
>>
>> As far as I can tell, the job finished successfully because I have the
>> processed output, and there are still 14 java threads that ducc user is
>> running.
>> The ducc monitory web server seems to be stuck as well, with the last
>> refresh showing the State as Running.
>>
>> Has anyone seen this before?
>>
>> Thanks!
>> Neal
>>
>>
>>
>
Re: Can not create new threads / ducc resource exhaustion while
running RawText Example
Posted by Eddie Epstein <ea...@gmail.com>.
There is a comment in the "Software Prerequisites" section but it is not
clear.
The docs will be changed to something like:
There are two user limits that may need changing for user ducc, preferably
in ducc's login profile. The default nfiles limit (-n) of 1024 will limit
the
number of broker connections. Raise this to 2x the number of user processes
running on the cluster. In some Linux OS the default max user process limit
(-u)
has been reduced to 1024. Raise this up to its hard limit.
Eddie
On Thu, Nov 7, 2013 at 8:32 AM, Eddie Epstein <ea...@gmail.com> wrote:
> Indeed, has been seen before. One interesting post that explains the
> problem is here:
>
> http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/
>
> I thought we updated the duccumentation to warn about this, but can't find
> it now.
> Will get that in now.
>
> Eddie
>
>
> On Wed, Nov 6, 2013 at 5:33 PM, Neal R Lewis <nr...@us.ibm.com> wrote:
>
>> I rebuilt DUCC yesterday, November 5th, 2013. I'm running the
>> DuccRawTextSpec.job with the same GutenBurg books, and
>> process_thread_count=24
>>
>> Machine: Fedora 19 x86_64. 32 Cores, 400GB Mem. Using OpenJDK.
>>
>>
>> Shortly after submitting the job, I get an: "OutOfMemoryError: unable to
>> create new native thread. "
>>
>> I can no longer run any commands as ducc (but can as root or a different
>> user) in the the shell, I get either:
>> "-bash: fork: retry: No child processes" OR
>> "-bash: fork: Resource temporarily unavailable"
>>
>> The ducc user become unusable.
>>
>> As far as I can tell, the job finished successfully because I have the
>> processed output, and there are still 14 java threads that ducc user is
>> running.
>> The ducc monitory web server seems to be stuck as well, with the last
>> refresh showing the State as Running.
>>
>> Has anyone seen this before?
>>
>> Thanks!
>> Neal
>>
>>
>>
>