You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Matthew F. Dennis (JIRA)" <ji...@apache.org> on 2010/06/09 21:49:15 UTC
[jira] Created: (CASSANDRA-1178) get_slice calls do not close files
when finished resulting in "too many open files" exceptions and rendering C
unusable
get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
-----------------------------------------------------------------------------------------------------------------------
Key: CASSANDRA-1178
URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.7
Reporter: Matthew F. Dennis
Assignee: Matthew F. Dennis
Fix For: 0.7
insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1178) get_slice calls do not close
files when finished resulting in "too many open files" exceptions and
rendering C unusable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877810#action_12877810 ]
Hudson commented on CASSANDRA-1178:
-----------------------------------
Integrated in Cassandra #462 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/462/])
fix FD leak. patch by mdennis; reviewed by jbellis for CASSANDRA-1178
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch, 0002-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1178) get_slice calls do not close files
when finished resulting in "too many open files" exceptions and rendering C
unusable
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1178:
-----------------------------------------
Attachment: 0001-trunk-1178.patch
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1178) get_slice calls do not close files
when finished resulting in "too many open files" exceptions and rendering C
unusable
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1178:
-----------------------------------------
Attachment: 0002-trunk-1178.patch
0002-trunk-1178.patch has suggested JavaDoc changes
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch, 0002-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1178) get_slice calls do
not close files when finished resulting in "too many open files" exceptions
and rendering C unusable
Posted by "Masood Mortazavi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877258#action_12877258 ]
Masood Mortazavi edited comment on CASSANDRA-1178 at 6/9/10 11:47 PM:
----------------------------------------------------------------------
Out of curiosity . . . What is the point, in this patch, in switching import statements around . . . Is there a policy to re-order them? . . . To the extent I can tell, the order of import statements is consistent with what one sees in the other files?
The rest of the fix seems OK to me as long as the "close" method in the IColumnIterator interface to SSTableSliceIterator is used properly to clean up resources . . . (although -- on a different plane -- objects that know, themselves, that they have nothing more to do are more friendly objects to use and harder to design) . . .
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1178) get_slice calls do
not close files when finished resulting in "too many open files" exceptions
and rendering C unusable
Posted by "Masood Mortazavi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877305#action_12877305 ]
Masood Mortazavi edited comment on CASSANDRA-1178 at 6/9/10 11:53 PM:
----------------------------------------------------------------------
Sounds reasonable . . .
It seems to me that he idea is that
(1) If you pass a non-null file object reference to the constructor, you are the best judge of when to close it.
(2) If you pass a null, instead, you will have to ask SSTableSliceIterator to clean up after you're done by calling "close" on it.
This is then the semantics of this class.
I think it may be useful to include this contractual fact in the javadoc for this class.
(Side note: It would probably be best not to allow IDE's to change the order of imports from the one that's common everywhere else unless there is a policy by this project to make such reordering. It is better to keep things consistent.)
was (Author: mortazavi):
Sounds reasonable . . .
It seems to me that he idea is that
(1) If you pass a non-null file object reference to the constructor, you are the best judge of when to close it.
(2) If you pass a null, instead, you will have to ask SSTableSlideIterator to clean up after you're done by calling "close" on it.
This is then the semantics of this class.
I think it may be useful to include this contractual fact in the javadoc for this class.
(Side note: It would probably be best not to allow IDE's to change the order of imports from the one that's common everywhere else unless there is a policy by this project to make such reordering. It is better to keep things consistent.)
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1178) get_slice calls do not close
files when finished resulting in "too many open files" exceptions and
rendering C unusable
Posted by "Masood Mortazavi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877305#action_12877305 ]
Masood Mortazavi commented on CASSANDRA-1178:
---------------------------------------------
Sounds reasonable . . .
It seems to me that he idea is that
(1) If you pass a non-null file object reference to the constructor, you are the best judge of when to close it.
(2) If you pass a null, instead, you will have to ask SSTableSlideIterator to clean up after you're done by calling "close" on it.
This is then the semantics of this class.
I think it may be useful to include this contractual fact in the javadoc for this class.
(Side note: It would probably be best not to allow IDE's to change the order of imports from the one that's common everywhere else unless there is a policy by this project to make such reordering. It is better to keep things consistent.)
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1178) get_slice calls do
not close files when finished resulting in "too many open files" exceptions
and rendering C unusable
Posted by "Masood Mortazavi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877305#action_12877305 ]
Masood Mortazavi edited comment on CASSANDRA-1178 at 6/9/10 11:54 PM:
----------------------------------------------------------------------
Sounds reasonable . . .
It seems to me that he idea is that
(1) If you pass a non-null file object reference to the constructor, you are the best judge of when to close it.
(2) If you pass a null, instead, you will have to ask SSTableSliceIterator to clean up after you're done by calling "close" on it.
This is then the semantics of this class when it comes to the file variable.
I think it may be useful to include this contractual/semantic fact in the javadoc for this class.
(Side note: It would probably be best not to allow IDE's to change the order of imports from the one that's common everywhere else unless there is a policy by this project to make such reordering. It is better to keep things consistent.)
was (Author: mortazavi):
Sounds reasonable . . .
It seems to me that he idea is that
(1) If you pass a non-null file object reference to the constructor, you are the best judge of when to close it.
(2) If you pass a null, instead, you will have to ask SSTableSliceIterator to clean up after you're done by calling "close" on it.
This is then the semantics of this class.
I think it may be useful to include this contractual fact in the javadoc for this class.
(Side note: It would probably be best not to allow IDE's to change the order of imports from the one that's common everywhere else unless there is a policy by this project to make such reordering. It is better to keep things consistent.)
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1178) get_slice calls do not close
files when finished resulting in "too many open files" exceptions and
rendering C unusable
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877272#action_12877272 ]
Matthew F. Dennis commented on CASSANDRA-1178:
----------------------------------------------
intellij often automatically reorders imports and removes unused ones (they call it "optimizing imports"). I just let it do it's thing in this regard (to be honest I've just gotten in the habit of ignoring the import section).
At least for get_slice, close is properly called (as evident by the patch fixing the problem). It may be the case that for other calls close is not correctly called, I didn't specifically look for such cases but 1) I haven't noticed a problem making other calls and 2) I didn't notice anything in the code while debugging this issue.
In general I agree that self contained objects are better but that doesn't necessarily fit in this case as the open file is really a system resource. Given the way Java does GC, such resources should be explicitly released.
I discussed this particular case with jbellis and the thought for the original patch that introduced this bug was that some callers would want to reuse an existing file. In such a case, they would be responsible for closing the file but that meant that SSTableSliceIterator shouldn't close it out from under them.
> get_slice calls do not close files when finished resulting in "too many open files" exceptions and rendering C unusable
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1178
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1178
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Matthew F. Dennis
> Assignee: Matthew F. Dennis
> Fix For: 0.7
>
> Attachments: 0001-trunk-1178.patch
>
>
> insert ~100K rows. Read them back in a loop. Notice "too many open files" exceptions in log. SSTableSliceIterator is never closing the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.