You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2012/09/20 11:58:07 UTC
[jira] [Created] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Simon Willnauer created LUCENE-4410:
---------------------------------------
Summary: Make FilteredQuery more flexible with regards to how filters are applied
Key: LUCENE-4410
URL: https://issues.apache.org/jira/browse/LUCENE-4410
Project: Lucene - Core
Issue Type: Improvement
Components: core/search
Affects Versions: 4.0-BETA
Reporter: Simon Willnauer
Priority: Blocker
Fix For: 5.0, 4.0
Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459522#comment-13459522 ]
Michael McCandless commented on LUCENE-4410:
--------------------------------------------
This sounds great! I'll look more at the patch ...
But surely this is not a blocker issue?
We (well, Robert: thanks!) are about to cut the final 4.0 release ... we shouldn't stuff features in at the last minute.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Priority: Blocker
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Attachment: LUCENE-4410.patch
I attached the wrong patch yesterday - thanks uwe for looking at it
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Priority: Major (was: Blocker)
s/blocker/major
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Attachment: LUCENE-4410.patch
This patch introduces a FilterStrategy similar to MTQ#RewriteMethod that wraps a given scorer & DocIDSet into a "Filtered Scorer". By default FilteredQuery uses the RandomAccessFilterStrategy that falls back to leap frog just like FQ did before. In contrast to the current FQ #useRandomAccess is now an impl detail of the RAFilterStrategy and might be removed with a different implementation that might rely on Scorer#cost() or something like that.
I also added a DocFirstFilterStrategy that applies filters only iff the delegate scorer matched. This seems more consistent with what other queries do (ie. MTQ) and provides more flexibility to the user
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Priority: Blocker
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459709#comment-13459709 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
Simon: I dont see the final methods in latest patch. Are you sure you uploaded the right one?
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459702#comment-13459702 ]
Michael McCandless commented on LUCENE-4410:
--------------------------------------------
I think we should target 4.1 for this? Ie commit to trunk, let it back, ship 4.0, backport.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460393#comment-13460393 ]
Simon Willnauer commented on LUCENE-4410:
-----------------------------------------
I committed this to trunk and added @lucene.experimental to FilterQuery#useRandomAccess on branch4x. I still think we should port the structure to 4x before we release but lets bake it in for a day or two and see how it goes. We are safe to change this in 4.1 now
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459721#comment-13459721 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
It is not biggish :-) It is same code as before :-) I don't care, but the RA autodetection was always horrible to me, now its hidden behind something else!
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer reassigned LUCENE-4410:
---------------------------------------
Assignee: Simon Willnauer
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459585#comment-13459585 ]
Simon Willnauer commented on LUCENE-4410:
-----------------------------------------
bq. I would aggree to this patch if the following is fixed:
I think that is a fair game. I think we should keep the DocFirst for trunk and if everybody agrees I merge the changes into 4.0 excluding the additional strategy. I already just copy pasted the current stuff in the the other two strategies. I can even remove the leap frog strategy in 4.x and keep in on trunk so I can use it to fallback to it in the DocFirst. Objections?
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459699#comment-13459699 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
The special PrimaryAdvancedLeapFrogScorer is final, once it is removed, we can make the original one final. For now I would make all methods except the primaryNext() one final (to help Hotspot).
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (LUCENE-4410) Make FilteredQuery more
flexible with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459556#comment-13459556 ]
Uwe Schindler edited comment on LUCENE-4410 at 9/20/12 11:34 PM:
-----------------------------------------------------------------
Hi,
I am also against rushing this in with 4.0. There is no slowdown in comparison to Lucene 3.6. My problems are also in the patch:
- I hate the crazy not-really-random access Bits impl in the DocFirstStrategy! It is definitely not random access, so it violates the contract. In this case the standard LeapFrog approach should be used (if not random access DocIdSet).
- I dont really see an improvement. The Bits interface is not allowed to throw IOException, so the filter should *only* return a Bits interface if its a very fast random access implementation. In all other cases the filter must not suport Bits and then LeapFrog has to be used.
- I would only see an improvement if the method Filter.getDocIdSet() is only called after the scorer hit the first document - but this does not work with random access. This was also not done in 3.6, so we should not rush.
- The ctor strategy param should never be NULL. The ctor without strategy should pass the explicit one down to this(..., Strategy)
was (Author: thetaphi):
Hi,
I am also against rushing this in with 4.0. There is no slowdown in comparison to Lucene 3.6. My problems are also in the patch:
- I hate the crazy not-really-random access Bits impl in the DocFirstStrategy! It is definitely not random access, so it violates the contract. In this case the standard LeapFrog approach should be used (if not random access DocIdSet).
- I dont really see an improvement. The Bits interface is not allowed to throw IOException, so the filter should *only* return a Bits interface if its a very fast random access implementation. In all other cases the filter must not suport Bits and then LeapFrog has to be used.
- I would only see an improvement if the method Filter.getDocIdSet() is only called after the scorer hit the first document - but this does not work with random access. This was also not done in 3.6, so we should not rush.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459541#comment-13459541 ]
Simon Willnauer commented on LUCENE-4410:
-----------------------------------------
bq. We (well, Robert: thanks!) are about to cut the final 4.0 release ... we shouldn't stuff features in at the last minute.
I marked this as a blocker since it really limits to a certain type of filters. I don't think this is a last minute feature really. I'd be totally happy to have only the structural refactoring in Lucene 4.0 and add the DocFirstStrategy later if that is consider a feature though.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459560#comment-13459560 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
In addition, stuff like this can also be added in later 4.x versions, as it does not change API of FilteredQuery, it just adds an additional ctor param
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459698#comment-13459698 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
Hi Simon, now I am happy. This is much better than before. I think this is a real improvement, I have no problem with getting that in. Maybe we commit it to trunk first and let it bake over the weekend.
PrimaryAdvancedLeapFrogScorer is a good workaround, it take some time until I understaood that this scorer simply "already knows" that it is already placed on the filter's first doc.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Fix Version/s: (was: 4.0)
4.1
I changed the fix version to 4.1 - I will keep this issue open until this is backported.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459627#comment-13459627 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
Hi,
I have one more comment for the DocFirst strategy:
The idea is good because it lets the query drive the collector and we only look up docs, the query hi (using the random access filter). This is sometimes better than passing it down as acceptdocs, because it would slowdown if the Bits interface is expensive and every query subclause must reevaluate the bits.get() method.
The problem I had with trhe patch is the crazy Bits implementation for the DocFirstStrategy, which had exactly this problem. Also it was not following the random access pattern, because it allowed the Bits.get() calls only in order. I can easily write a BooleanScorer1-like query that violates this (because a query with more than one sub-clause can easily call Bits.get() out of order for each sub-clause).
The DocFirstStrategy wants the query drive the collection, so the non-bits approach should either use LeapFrog (which may be expensive if the filter has ineffective nextDoc()) or it should also implemen DocFirst in order. I would rename that strategy to QueryFirstStrategy and make 2 scorers for it:
- a random access one calling Bits.get() for every hit of the query
- a sequential one that calls nextDoc() only on the Query, never on the filter. The filter is only advanced to the current query doc. By this the filter oly scans through its docs very seldom (when there is no hit after advance).
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459715#comment-13459715 ]
Michael McCandless commented on LUCENE-4410:
--------------------------------------------
bq. We can only do this, if we add "experimental" to the random access detetion method...
+1
I just don't think we should be making such biggish changes just before releasing 4.0...
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459686#comment-13459686 ]
Simon Willnauer commented on LUCENE-4410:
-----------------------------------------
For the given patch I'd commit that to trunk and backport the FilterStrategy structure without the enhancements to 4.0. That way we only have the API change in 4.0 and no new feature but that would allow us to add all the other stuff in this patch to go into 4.1 without breaking anything.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459556#comment-13459556 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
Hi,
I am also against rushing this in with 4.0. There is no slowdown in comparison to Lucene 3.6. My problems are also in the patch:
- I hate the crazy not-really-random access Bits impl in the DocFirstStrategy! It is definitely not random access, so it violates the contract. In this case the standard LeapFrog approach should be used (if not random access DocIdSet).
- I dont really see an improvement. The Bits interface is not allowed to throw IOException, so the filter should *only* return a Bits interface if its a very fast random access implementation. In all other cases the filter must not suport Bits and then LeapFrog has to be used.
- I would only see an improvement if the method Filter.getDocIdSet() is only called after the scorer hit the first document - but this does not work with random access. This was also not done in 3.6, so we should not rush.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459583#comment-13459583 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
I would aggree to this patch if the following is fixed:
- remove DocFirst* for now. See above for reasons.
- only copy-paste current code in the new API
- make the ctor with strategy not accept null, require it to pass an explicit strategy. The ctor without strategy should pass the explicit one down to this(..., Strategy)
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459545#comment-13459545 ]
Robert Muir commented on LUCENE-4410:
-------------------------------------
I don't agree with blocker either. Should it block 3.6.2 as well? 4.0 is significantly more flexible than 3.x was.
3.x only had one way to execute a filter: leapfrog.
4.x has three ways: leapfrog, bits, and auto. the default is auto.
{code}
is.search(new FilteredQuery(q,f) {
protected boolean useRandomAccess(Bits bits, int firstFilterDoc) {
return false; // always leapfrog: act like 3.x
}
});
...
is.search(new FilteredQuery(q,f) {
protected boolean useRandomAccess(Bits bits, int firstFilterDoc) {
return true; // never leapfrog if Bits are available
}
});
{code}
Separately (unrelated to release management) I like this idea and I think we should do it.
But as i said on the mailing list, i think we should be tackling bugs, javadocs, and tests and approaching stability.
it makes me nervous to add more features to our filter execution right before release.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Attachment: LUCENE-4410.patch
here is a new patch that uses a QueryFirstLeapFrog if we can't pull bits in the DocFirstStrategy. I generalized the LeapFrog scorer such that it can do query first, filter first and "filter already advanced" I also fixed the parameters in the ctor and added both QUERY_FIRST and FILTER_FIRST strategies to FilterQuery. I hooked that up with Random testing and all tests pass
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved LUCENE-4410.
-------------------------------------
Resolution: Fixed
committed to 4.x in revision 1389462.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459703#comment-13459703 ]
Uwe Schindler commented on LUCENE-4410:
---------------------------------------
We can only do this, if we add "experimental" to the random access detetion method... Otherwise we are in backwards-compatibility hell in 4.1
I checked the code, it's identical to the one before, just class hierarchy changed... We can maybe only remove the new QueryFirstStrategy, but the other one is 100% identical to current 4.0 code. I see no problem with it going in.
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4410) Make FilteredQuery more flexible
with regards to how filters are applied
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4410:
------------------------------------
Attachment: LUCENE-4410.patch
next iteration, fixing some javadoc issues and adding a changes entry. I also made all methods that should not be subclassed final on LeapFrogScorer so hotspot doesn´t mess around with it. I think its ready. If nobody objects I will commit this to trunk tomorrow morning
> Make FilteredQuery more flexible with regards to how filters are applied
> ------------------------------------------------------------------------
>
> Key: LUCENE-4410
> URL: https://issues.apache.org/jira/browse/LUCENE-4410
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Affects Versions: 4.0-BETA
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4410.patch, LUCENE-4410.patch, LUCENE-4410.patch
>
>
> Currently FilteredQuery uses either the "old" lucene 3 leap frog approach or pushes the filter down together with accepted docs. Yet there might be more strategies required to fit common usecases like geo-filtering where a rather costly function is applied to each document. Using leap frog this might result in a very slow query if the filter is advanced since it might have linear running time to find the next valid document. We should be more flexible with regards to those usecases and make it possible to either tell FQ what to do or plug in a strategy that applied a filter in a different way.
> The current FQ impl also uses an heuristic to decide if RA or LeapFrog should be used. This is really an implementation detail of the strategy and not of FQ and should be moved out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org