You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Yonik Seeley (JIRA)" <ji...@apache.org> on 2007/03/03 19:20:50 UTC
[jira] Created: (LUCENE-821) single norm file still uses up
descriptors
single norm file still uses up descriptors
------------------------------------------
Key: LUCENE-821
URL: https://issues.apache.org/jira/browse/LUCENE-821
Project: Lucene - Java
Issue Type: Bug
Reporter: Yonik Seeley
Assigned To: Yonik Seeley
The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12477706 ]
Michael McCandless commented on LUCENE-821:
-------------------------------------------
Whoa, you beat me to it: this was the next bug I was about to open!
And I had worked through a patch, even setting in = null in the newly
added Norm.close method :)
And then also re-discovering that TestMultiSearcher closes its
searcher and then keeps using it...
Your patch looks perfect to me. +1
I like the removal of the .clone() since norms(...) is synchronized.
The one thing I was still trying to work out is if we could somehow
close the singleNormStream once all Norm instances that share it had
cached. It seems like a nice to have, but, since norms(...) is
already synchronized we could have a simple refcount to track how many
Norm instances still required it and then close the stream when that
hits 0? This way we can free up 1 more file descriptor in certain
cases.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12477759 ]
Michael McCandless commented on LUCENE-821:
-------------------------------------------
> The reason is predictability... since you don't know what queries are coming, you don't know when that descriptor will be closed, and hence need to plan for the upper bound on descriptor usage anyway.
Agreed, the upper bound is what counts here, so it's not much help to reduce by one at some random later time.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12477761 ]
Doron Cohen commented on LUCENE-821:
------------------------------------
wow.. I totally missed this point in 756. Fix here seems perfect to me too. Thanks for catching and fixing this Yonik.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12477676 ]
Yonik Seeley commented on LUCENE-821:
-------------------------------------
> - removed extra IndexInput clone() when reading the norms... why was this there originally?
Going back to the original SegmentReader implementation (2001), getNorms() wasn't synchronized, hence the clone of IndexInput was needed.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated LUCENE-821:
--------------------------------
Attachment: norms.patch
Attaching patch.
Changes:
- Only open a single IndexInput for the .nrm file and reuse it for all
norms sharing that file in the segment. No clone is needed since
only a single norm is read at any time.
- close norms when no longer needed (idea from Michael)
- removed extra IndexInput clone() when reading the norms... why was this there originally?
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12477733 ]
Yonik Seeley commented on LUCENE-821:
-------------------------------------
Yeah, I considered refcounting the input, but I'm not sure that it's worth it.
The reason is predictability... since you don't know what queries are coming, you don't know when that descriptor will be closed, and hence need to plan for the upper bound on descriptor usage anyway.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-821) single norm file still uses up
descriptors
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley resolved LUCENE-821.
---------------------------------
Resolution: Fixed
committed. Thanks for the review Mike.
> single norm file still uses up descriptors
> ------------------------------------------
>
> Key: LUCENE-821
> URL: https://issues.apache.org/jira/browse/LUCENE-821
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assigned To: Yonik Seeley
> Attachments: norms.patch
>
>
> The new index file format with a single .nrm file for all norms does not decrease file descriptor usage.
> The .nrm file is opened once for each field with norms in the index segment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org