Re: Too many files open after upgrading to 10.8

[Kristian Waagan <kr...@oracle.com>]

On 31.01.2012 19:36, Katherine Marsden wrote:
> I have few details yet, but have a report from a user upgrading from 
> 10.5 to 10.8 who is reporting occurrences of  IOException too many 
> open files on both linux and windows in their test suite after the 
> upgrade.  The actual error is not usually in the Derby code but  it 
> may be that Derby is hogging too many files causing this failure later 
> in their code.
>
> They can reproduce the problem in a 15 minute test.  Is there a 
> generic java or operating system way to trace the opening and closing 
> of files?  If not, what would be the best place  to add tracing to Derby?

Hi Kathey,

I once tried to add tracing to Derby (using dynamic byte code 
instrumentation) for this, but found it somewhat hard to get it working 
well enough. At this stage, I'd go for operating system tools, for 
instance 'lsof' on Linux and the resource manager available on Windows 
(depends on Windows version).

Would also be helpful to know the limit they're hitting (i.e. ulimit -n).


Regards,
-- 
Kristian

>
> Thanks
>
> Kathey
>
>

Re: Too many files open after upgrading to 10.8

[Kristian Waagan <kr...@oracle.com>]

On 01.02.2012 22:06, Katherine Marsden wrote:
> On 2/1/2012 3:13 AM, Kristian Waagan wrote:
>>
>> Hi Kathey,
>>
>> I once tried to add tracing to Derby (using dynamic byte code 
>> instrumentation) for this, but found it somewhat hard to get it 
>> working well enough. At this stage, I'd go for operating system 
>> tools, for instance 'lsof' on Linux and the resource manager 
>> available on Windows (depends on Windows version).
>>
> Thanks Kristian, we'll stick with the os tools for starters to try to 
> figure out what is going on.
>
>> Would also be helpful to know the limit they're hitting (i.e. ulimit 
>> -n).
>>
>
> I am waiting to hear back on experiments to see what files are open, 
> but my understanding is that their tests are using the default so 1024 
> per process on Linux.  We usually recommend upping that, but it makes 
> sense that they would like to understand what has changed with the 
> 10.8 upgrade.  One thing I have noticed that every classapath entry, 
> internal to the jre and user specified has its own open file. In my 
> environment, which just has a few other things besides Derby, it 
> really adds up with the IBM 1.6 JVM.
> 166 total,  112 for the java classes, 22 for the derby jars including 
> Locales, 13 os  libraries and locale  stuff and the rest my other 
> classpath entries.   The user is seeing 512 open just  with their 
> launcher that sets the classpath before doing anything.
>
> One thing Mike mentioned is that it is possible that the istat daemon 
> is opening files and pushing them over the edge.  Not a bug but just 
> an increase if there are a lot of tables to be updated.

That may be, but the istat daemon only operates on conglomerates being 
accessed already by the user. I don't know exactly how this would work 
if a lot of work piles up, the daemon is storing a TableDescriptor for 
each unit of work. In any case, the queue is capped so it shouldn't grow 
that much.
Another theory is that something isn't being cleaned up.

Will be interesting to see the list of open handles.


-- 
Kristian

>
> Kathey
>
>
>
>
>
>
>
>
>>
>> Regards,
>

Re: Too many files open after upgrading to 10.8

[Katherine Marsden <km...@sbcglobal.net>]

On 2/1/2012 3:13 AM, Kristian Waagan wrote:
>
> Hi Kathey,
>
> I once tried to add tracing to Derby (using dynamic byte code 
> instrumentation) for this, but found it somewhat hard to get it 
> working well enough. At this stage, I'd go for operating system tools, 
> for instance 'lsof' on Linux and the resource manager available on 
> Windows (depends on Windows version).
>
Thanks Kristian, we'll stick with the os tools for starters to try to 
figure out what is going on.

> Would also be helpful to know the limit they're hitting (i.e. ulimit -n).
>

I am waiting to hear back on experiments to see what files are open, but 
my understanding is that their tests are using the default so 1024 per 
process on Linux.  We usually recommend upping that, but it makes sense 
that they would like to understand what has changed with the 10.8 
upgrade.  One thing I have noticed that every classapath entry, internal 
to the jre and user specified has its own open file. In my environment, 
which just has a few other things besides Derby, it really adds up with 
the IBM 1.6 JVM.
166 total,  112 for the java classes, 22 for the derby jars including 
Locales, 13 os  libraries and locale  stuff and the rest my other 
classpath entries.   The user is seeing 512 open just  with their 
launcher that sets the classpath before doing anything.

One thing Mike mentioned is that it is possible that the istat daemon is 
opening files and pushing them over the edge.  Not a bug but just an 
increase if there are a lot of tables to be updated.

Kathey








>
> Regards,