You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Paolo Castagna (JIRA)" <ji...@apache.org> on 2011/09/03 01:32:09 UTC
[jira] [Created] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Optimise ORDER BY + OFFSET + LIMIT queries
------------------------------------------
Key: JENA-109
URL: https://issues.apache.org/jira/browse/JENA-109
Project: Jena
Issue Type: Improvement
Components: ARQ
Reporter: Paolo Castagna
Priority: Minor
The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
Maybe we can assume users will not hit 'next page' too many times.
We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Paolo Castagna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096943#comment-13096943 ]
Paolo Castagna commented on JENA-109:
-------------------------------------
> Setting the window to 100000 interacts with JENA-44, which gives parallelism in the sort. I suggest a more cautious limit of a few hundred items.
Ok.
Even if the interaction is one exclude the other rather than causing problems or wrong results.
Maybe, the window for JENA-89 and JENA-109 should be configurable via ARQ symbols as it will be the threshold for JENA-44. This way, users can tune their systems as they wish. Not an excuse to come up with sensible and reasonable defaults though.
> There's no need to change (TopN). [...] put a slice over the "top"
Ok, it's much simpler this way. I'll do that.
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Assignee: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Paolo Castagna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paolo Castagna resolved JENA-109.
---------------------------------
Resolution: Fixed
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Assignee: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Paolo Castagna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paolo Castagna closed JENA-109.
-------------------------------
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Assignee: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Paolo Castagna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paolo Castagna reassigned JENA-109:
-----------------------------------
Assignee: Paolo Castagna
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Assignee: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Paolo Castagna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paolo Castagna updated JENA-109:
--------------------------------
Attachment: JENA-109_ARQ_r1164652.patch
A little bit more testing is necessary, however this shows my proposal.
What do you think?
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-109) Optimise ORDER BY + OFFSET + LIMIT
queries
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096919#comment-13096919 ]
Andy Seaborne commented on JENA-109:
------------------------------------
It's a reasonable thing to add.
Setting the window to 100000 interacts with JENA-44, which gives parallelism in the sort. I suggest a more cautious limit of a few hundred items. Some performance comparision can then be done.
About the patch:
There's no need to change (TopN). Instead, rewrite the query to have a slice and
SELECT ..{ } ORDER BY ... OFFSET X LIMIT N
(slice X _
(top (N ...)
(...)
))
that is, put a slice over the "top" and apply the top to N+X. This keeps (top) focused on what it does - a fixed length sort/limit.
i.e.
SELECT *
{ SELECT ... { } ORDER BY ... LIMIT N+X }
OFFSET X
There's no need for an outer LIMIT (which would be N)
> Optimise ORDER BY + OFFSET + LIMIT queries
> ------------------------------------------
>
> Key: JENA-109
> URL: https://issues.apache.org/jira/browse/JENA-109
> Project: Jena
> Issue Type: Improvement
> Components: ARQ
> Reporter: Paolo Castagna
> Assignee: Paolo Castagna
> Priority: Minor
> Labels: optimization, sparql
> Attachments: JENA-109_ARQ_r1164652.patch
>
>
> The benefits of JENA-89 gets lost as soon as someone uses OFFSET, even for low values.
> Maybe we can assume users will not hit 'next page' too many times.
> We can increase the TOPN_LIMIT_THRESHOLD in TransformTopN to 100000 (?) and apply the same TopN optimization we discussed in JENA-89 even when we have OFFSET (when OFFSET + LIMIT < TOPN_LIMIT_THRESHOLD).
> In QueryIterTopN we simply return IteratorArray.create(y, offset, offset+limit) instead of IteratorArray.create(y).
> This way we can avoid total sort for a few number of small pages (all within the 100000 threshold).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira