You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2012/11/05 13:16:12 UTC
[jira] [Created] (LUCENE-4538) Cache DocValues DirecSource
Simon Willnauer created LUCENE-4538:
---------------------------------------
Summary: Cache DocValues DirecSource
Key: LUCENE-4538
URL: https://issues.apache.org/jira/browse/LUCENE-4538
Project: Lucene - Core
Issue Type: Improvement
Components: core/codecs
Affects Versions: 4.0
Reporter: Simon Willnauer
Fix For: 4.1, 5.0
Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490627#comment-13490627 ]
Robert Muir commented on LUCENE-4538:
-------------------------------------
I haven't looked in detail (will try to do a real review later) but I think something like this is really needed.
Its currently absurd: we make it easy as pie to load up enormous data structures in RAM but nearly impossible to use stuff from disk.
It should be the other way around.
{quote}
We can certainly reduce the overhead (maybe different issue)
{quote}
We should do this too!
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4538:
------------------------------------
Attachment: LUCENE-4538.patch
a patch including loadDirectSource in the SourceCache caching DirectSources in a CloseableThreadLocal
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4538:
------------------------------------
Attachment: LUCENE-4538.patch
here is a new patch makeing the loadSource & loadDirectSource protected. This is really confusing if you have two ways to get a Source instance and you need to take care of if it is cached or not. This really should not have been public at all.
I will commit this soon
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch, LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490643#comment-13490643 ]
Robert Muir commented on LUCENE-4538:
-------------------------------------
Among other things: look at how expensive it is today to create a Packed ints direct source (its just an example, other DV impls also read lots of similar stuff)
# IndexInput.clone(): PackedIntValues:221 return new
PackedIntsSource(datIn.clone(), true);
# readLong(): PackedIntValues:223 minValue = dataIn.readLong();
# readLong(); PackedIntValues:224 defaultValue = dataIn.readLong();
# readInt(); Codec magic <-- PackedInts.getDirectReader()
# readString(); Codec name
# readInt(); Codec version
# readVInt(); bits per value
# readVInt(); value count
# readVint(); format
I opened LUCENE-4539 for this. Taking a cursory look at e.g. the packedints impl, i think we can fix
it by just fixing our impl (e.g. no file formats change).
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491558#comment-13491558 ]
Robert Muir commented on LUCENE-4538:
-------------------------------------
Thanks Simon!
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch, LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved LUCENE-4538.
-------------------------------------
Resolution: Fixed
committed to trunk in rev. 1406153
backported to 4.x in rev. 1406169
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch, LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490638#comment-13490638 ]
Adrien Grand commented on LUCENE-4538:
--------------------------------------
bq. We can certainly reduce the overhead (maybe different issue)
Are you speaking about the PackedInts headers?
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-4538) Cache DocValues DirecSource
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer reassigned LUCENE-4538:
---------------------------------------
Assignee: Simon Willnauer
> Cache DocValues DirecSource
> ----------------------------
>
> Key: LUCENE-4538
> URL: https://issues.apache.org/jira/browse/LUCENE-4538
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4538.patch, LUCENE-4538.patch
>
>
> Currently the user need to make sure that a direct source is not shared between threads and each time someone calls getDirectSource we create a new source which has a reasonable overhead. We can certainly reduce the overhead (maybe different issue) but it should be easier for the user to get a direct source and handle it. More than that is should be consistent with getSource / loadSource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org