You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2011/06/12 19:11:51 UTC
[jira] [Created] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
MockDirectoryWrapper should throw 'too many open files' if too many are open
----------------------------------------------------------------------------
Key: LUCENE-3195
URL: https://issues.apache.org/jira/browse/LUCENE-3195
Project: Lucene - Java
Issue Type: Test
Reporter: Robert Muir
regardless of operating system.
We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048381#comment-13048381 ]
Uwe Schindler commented on LUCENE-3195:
---------------------------------------
Great idea!
I dont think it will help on Jenkins (as it was caused by something else), but its a good idea.
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048377#comment-13048377 ]
Shai Erera commented on LUCENE-3195:
------------------------------------
That's a great idea !
bq. and it should throw exception on openInput
and createOutput too?
bq. we should take a good default value
Let's just start w/ 1024? I doubt our tests need to open that many anyway, and it's, I think, low enough to fit any OS?
Also, and this can be added in future iterations, I think it'd be good if the exception includes a list of all test classes + count of open files, to make it easier to spot offending classes.
Hmmm ... now that I think of it, if our tests are run in different JVMs, we will fail to faithfully track the number of open file handles (I mean the *true* number)? In that case, we might want to lower the value to whatever we think a single JVM should allocate ... which gets tricky.
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048389#comment-13048389 ]
Shai Erera commented on LUCENE-3195:
------------------------------------
bq. Well, the test spits this out when it fails already? it says "tests run in this jvm: [TestX, TestY, TestZ]..."
Ok. But for easier spotting the offending test, I meant it'd be more useful if at the point of exception you'd know which tests keep the open files and how many each test holds. Because "TestX" and "TestY" which run in this JVM may not keep any open files at the moment, so it's just noise that you need to filter? Are they reported even if they have already finished?
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048379#comment-13048379 ]
Robert Muir commented on LUCENE-3195:
-------------------------------------
{quote}
Also, and this can be added in future iterations, I think it'd be good if the exception includes a list of all test classes + count of open files, to make it easier to spot offending classes.
{quote}
Well, the test spits this out when it fails already? it says "tests run in this jvm: [TestX, TestY, TestZ]..."
{quote}
Hmmm ... now that I think of it, if our tests are run in different JVMs, we will fail to faithfully track the number of open file handles (I mean the true number)? In that case, we might want to lower the value to whatever we think a single JVM should allocate ... which gets tricky.
{quote}
Right, its not going to be totally accurate anyway, I guess I think the goal is not to emulate an operating system but instead to catch a runaway test on all platforms.
For example, as a start MDW could just check its own hashmap, as 99% of tests only make one directory, and a JVM only runs one test at a time.
But it would be better for MDW to have a static atomic counter, for better detection of the crazy tests (those are probably the ones opening too many files sometimes anyway).
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048401#comment-13048401 ]
Shai Erera commented on LUCENE-3195:
------------------------------------
bq. so that if one test blows it up, the rest are @Ignored?
That sounds good.
bq. They had better not! or, they should fail because they had file leaks!
Right :). But I meant what if N tests are run in parallel in the same JVM (do we do that today?), and one test causes the TooManyOpenFiles exception (or FNFE as you said), it'd be good to know which how many open files each test holds.
If tests are run sequentially in the same JVM though, then this is unnecessary and we can do w/ that global counter + assume().
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048393#comment-13048393 ]
Robert Muir commented on LUCENE-3195:
-------------------------------------
but in line with what you said Shai, i think its hard to debug when one test goes over the limit, and fails, currently its like this chain reaction where all the tests in hudson then fail.
so if we add this atomic variable, maybe we check it with an assume() in beforeClass or something, so that if one test blows it up, the rest are @Ignored?
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3195) MockDirectoryWrapper should throw
'too many open files' if too many are open
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048392#comment-13048392 ]
Robert Muir commented on LUCENE-3195:
-------------------------------------
{quote}
Because "TestX" and "TestY" which run in this JVM may not keep any open files at the moment, so it's just noise that you need to filter?
{quote}
They had better not! or, they should fail because they had file leaks!
I think its useful to provide some debugging information, but on the other hand I also think the exception we throw should be as much as possible, exactly like the one the OS throws (isn't it FNFE or something strange?)
> MockDirectoryWrapper should throw 'too many open files' if too many are open
> ----------------------------------------------------------------------------
>
> Key: LUCENE-3195
> URL: https://issues.apache.org/jira/browse/LUCENE-3195
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Robert Muir
>
> regardless of operating system.
> We should take a good default value, like the lowest one we consider 'reasonable' macosX? and it should throw exception on openInput if openFiles hashmap is too large.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org