You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Simon Helsen (JIRA)" <ji...@apache.org> on 2012/08/13 21:00:40 UTC
[jira] [Created] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Simon Helsen created JENA-294:
---------------------------------
Summary: TransformFilterEquality does not handle starting OPTIONAL well
Key: JENA-294
URL: https://issues.apache.org/jira/browse/JENA-294
Project: Apache Jena
Issue Type: Bug
Components: ARQ
Affects Versions: Jena 2.7.4
Reporter: Simon Helsen
There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438966#comment-13438966 ]
Simon Helsen commented on JENA-294:
-----------------------------------
yes, I saw this. Ok, in this case, it would be trivial since
SELECT *
{
?x :p ?o2
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
}
is equivalent to
SELECT *
{
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
?x :p ?o2
}
(I can provide a patch, but does that help here?)
Ok, I see the tests in TestFilterTransform. I assume the exact test for this particular use-case depends a bit on how it is fixed. One possibility (but I don't know if that could affect performance in other ways) is to always move all non-OPTIONAL constraints before OPTIONAL constraints and then let TransformFilterEquality do its normal job.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435959#comment-13435959 ]
Simon Helsen commented on JENA-294:
-----------------------------------
Andy, I am confused by you answer "yes, it does". Are you saying that the proposed patch works correctly for that case? And if so, are you saying that
(prefix ((: <http://example/>))
(assign ((?x :x))
(conditional
(conditional
(table unit)
(bgp (triple :x :r ?s)))
(bgp (triple :x :q ?o)))))
is a correct plan?
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438829#comment-13438829 ]
Simon Helsen commented on JENA-294:
-----------------------------------
right, the problem is that if the filters with the ?R=<uri> pattern are not converted to assigns, we end up with potentially countless of returns which is why we got the performance breakdown from a few ms to minutes. With the assigns, the variable is not expanded and tested each time. What you suggest makes sense. You introduce an assign for each occurrence of the variable (deep down) and just keep the filter where it was originally to make sure the semantics is correct (i.e. as before the optimization). I'll definitely check it out
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434589#comment-13434589 ]
Simon Helsen commented on JENA-294:
-----------------------------------
ah, I see now, it seems that the expansion (substitute) should take place regardless, but the assignment should only happen if the conditional evaluates to something. So it should be pushed inside and I see now that this is what you were saying in one of the comments above.
Btw, the reason our example does not fail is that the WHERE clause also has a non-optional condition on the filter variable, so even if the optional does not match, the assignment is always correct.
I am attaching a new patch. If an assignment sits on a conditional or leftJoin, it shifts it inside the RHS. Now, I must say I am not entirely sure how nested conditionals are evaluated, so perhaps this is not quite right. E.g. I am wondering if the assignment should also be pushed into the LHS and then recursively down. The substitution itself is not affected I'd say and the precondition check remains the same as well because a LHS in these left joins which is a unit does not.
So, e.g. query
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :p ?z }
OPTIONAL { ?y :q ?z }
FILTER(?z = :a)
}
turns into
(prefix ((: <http://example/>))
(conditional
(conditional
(table unit)
(bgp (triple ?x :p :a)))
(assign ((?z :a))
(bgp (triple ?y :q :a)))))
but query
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :p ?z }
OPTIONAL { ?y :q ?z }
?w :r ?z
FILTER(?z = :a)
}
(which is more like our use case right now) produces this plan:
(prefix ((: <http://example/>))
(assign ((?z :a))
(sequence
(conditional
(conditional
(table unit)
(bgp (triple ?x :p :a)))
(bgp (triple ?y :q :a)))
(bgp (triple ?w :r :a)))))
and your example
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
}
produces plan
(prefix ((: <http://example/>))
(conditional
(table unit)
(assign ((?x :x))
(bgp (triple :x :q ?o)))))
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434398#comment-13434398 ]
Simon Helsen commented on JENA-294:
-----------------------------------
right, but because we are under time pressure (internally) I need at least a probable correct solution for the simple case which was affecting us. I am not quite following your statement "It may not be necessary, or it may be simpler, after leftjoin to condition as the precondition for that rewrite depends on scoping of variables as well"
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438956#comment-13438956 ]
Simon Helsen commented on JENA-294:
-----------------------------------
While I agree your version may be "more" conventional, we had clients who produced the one I provided. Not only that, their "less conventional query" was generated by a generator so I am sure they had their reasons. Finally, and this is important to us, they had their queries tested for performance on our previous adoption of 2.6.x and the move to 2.7.x turned it into a performance regression for them. As I have mentioned before, we are talking a few 100ms versus many minutes.
So, in this case, we really need a fix for the query I provided above. Otherwise, we are still in the same situation as before I opened this defect (which is why I reopened). More generally, I am not sure why the same "trick" cannot be applied here. Leave filter as it is and simply push the assignment down. Why is that unique to OPTIONAL?
As for providing tests for jena-arq/testing/ARQ/OptFilterEquality/, I am happy to add a test if I have one, but it seems that this test suite is only to check query correctness not whether the optimization took place, or am I missing something
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne resolved JENA-294.
--------------------------------
Resolution: Fixed
Fix Version/s: ARQ 2.9.4
Assignee: Andy Seaborne
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434910#comment-13434910 ]
Andy Seaborne commented on JENA-294:
------------------------------------
The new patch causes errors in the test suite.
I constructed this example where the ?x :p ?o part does match the data.
Data:
:x :p 123 .
Query:
SELECT *
{
?x :p ?o .
OPTIONAL { ?x :q1 ?o }
FILTER(?x = :x)
}
gives the wrong answers. If the left side matches and the right side does not, then no assignment is done but the (bgp (triple :x :p ?o)) has been rewritten.
(conditional
(bgp (triple :x :p ?o))
(assign ((?x :x))
(bgp (triple :x :q1 ?o)))))
The problem is that pushing the assign in always means that the fixed (left) part of the optional misses it. A hack to push if and only if the left is a unit table works in this one example but I'm not sure about more complex cases yet.
It is essential we have a test suite so we can be sure we do not end up breaking existing correct behavior. I'll create one.
Pushing inwards when the other side has unrelated variables would generalize the optimization (which, incidentally, would be good for filters as well).
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434106#comment-13434106 ]
Andy Seaborne edited comment on JENA-294 at 8/15/12 12:23 AM:
--------------------------------------------------------------
Example:
data:
@prefix : <http://example/> .
:s :q 123 .
query:
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :p ?z }
FILTER(?z = :a)
}
result:
No rows : The OPTIONAL does not match, so the result to that point is one row, not bindings. The FILTER fails, ?z is not bound. Hence result is no rows.
With the patch (the change around line 132 is the critical point):
One row, ?z bound to :a caused by assign to the empty row.
Checking, then pushing the assign inside the optional right-hand side should address this making the assignment happen if and only if the optional clause matches.
was (Author: andy.seaborne):
Example:
data:
@prefix : <http://example/> .
:s :q 123 .
query:
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :p ?z }
FILTER(?z = :a)
}
result:
No rows : The OPTIONAL does not match, so the result to that point is one row, not bindings. The FILTER fails, ?z is not bound. Hence result is no rows.
With the patch (the change around line 132 is the critical point):
One row, ?z bound to :a caused by assign to the empty row.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne updated JENA-294:
-------------------------------
Attachment: test-filter-equality.zip
Some tests for filter equality. These are a test suite runnable by arq.qtest. These are functional tests, they do not check optimization has been applied; they check for correct query results.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Helsen updated JENA-294:
------------------------------
Attachment: patch.txt
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433427#comment-13433427 ]
Simon Helsen commented on JENA-294:
-----------------------------------
this is related to JENA 283
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435500#comment-13435500 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Yes, it does.
Tested to cover that.
The tests are in
https://svn.apache.org/repos/asf/jena/Scratch/AFS/Jena-Dev/trunk/FilterEquality/
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438964#comment-13438964 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Yes - OptFilterEquality checks correctness (optimization is in TestFilterTransform) but we need tests in this part of the suite
1/ to get proper coverage of all the case you want considered.
2/ to ensure optimization does not change results.
So they are "necessary" tests - not "sufficient" tests.
They are short scripted tests using the SPARQL-WG manifest format (not in java).
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435436#comment-13435436 ]
Simon Helsen commented on JENA-294:
-----------------------------------
When I apply your patch to the trunk and try it on the following query:
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :r ?s }
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
}
I get the following plan:
(prefix ((: <http://example/>))
(assign ((?x :x))
(conditional
(conditional
(table unit)
(bgp (triple :x :r ?s)))
(bgp (triple :x :q ?o)))))
Does this not exhibit the same problem as discussed earlier? I.e. the bindings on ?x for :r and :q may not exist, yet a row would be returned with a binding for :x
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne updated JENA-294:
-------------------------------
Attachment: jena-294-afs-1.patch
Illustrative patch (this is relative to the current SVN trunk - there has been some tidying up as a side-effect of working on this e..g remove Op2.setRight -- mutating ops can cause problems; use .copy(,) )
The patch shows a possible solution. It is not tested enough to commit to the main codeline.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438822#comment-13438822 ]
Andy Seaborne commented on JENA-294:
------------------------------------
I discovered a completely different approach which pushes in assigns but also retains the filter in case it is still needed. The processing is still much more efficient. Filters are quite cheap to execute - the optimization is to reduce the unnecessary returns from the sub-expressions.
Please try out tonight's dev build.
For the last example:
(filter (= ?x :x)
(conditional
(conditional
(table unit)
(assign ((?x :x))
(bgp (triple :x :q ?o))))
(assign ((?x :x))
(bgp (triple :x :r ?s)))))
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Helsen reopened JENA-294:
-------------------------------
I tested against HEAD and yes, for queries with only OPTIONALs,everything seems to work correctly now, but the optimization is lost whenever the query starts with OPTIONAL, but also has a non-optional clause. E.g.
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
?x :p ?o2
}
produces
(prefix ((: <http://example/>))
(filter (= ?x :x)
(sequence
(conditional
(table unit)
(bgp (triple ?x :q ?o)))
(bgp (triple ?x :p ?o2)))))
The assigns are lost
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434066#comment-13434066 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Good idea although it's a bit more tricky for a complex left-hand side of optional.
Two general cases to be dealt with: the ubiquitous, doubly nested optional:
OPTIONAL {
... no use of ?y ....
OPTIONAL { ... uses ?y ... }
}
FILTER (?y = <x>)
and a specific case of ...
OPTIONAL {
... does not match ...
}
FILTER (?y = <x>)
I wonder if the right thing to do if push the assignment itself inside the OPTIONAL right hand side. With a unit left handside, this is possible - the generalisation (needs testing) is that the LHS does not use the filter variable.
Doubly nested optionals can only be transformed this way if the they can be converted to (conditional) later. Therefore either reorder the optimize so the (conditional) test is first and be more aggressive on condition or expose the test in some way.
The optimizer tests are in TestOptimizer
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451979#comment-13451979 ]
Simon Helsen commented on JENA-294:
-----------------------------------
although I would like this optimization problem be fixed at some point, we managed to get our client to change their query generator to bypass the problem. We are not aware of anyone else relying on the pattern, so for us the prioity is much lower now. It should certainly not inhibit a new release
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: filter-equality-13.rq, filter-equality-13.srj, jena-294-afs-1.patch, manifestPatchForOptFilterEquality.txt, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435237#comment-13435237 ]
Simon Helsen commented on JENA-294:
-----------------------------------
thanks Andy. I'll take a look at it and also test it locally within our framework, even though those queries are more conservative
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434106#comment-13434106 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Example:
data:
@prefix : <http://example/> .
:s :q 123 .
query:
PREFIX : <http://example/>
SELECT *
{
OPTIONAL { ?x :p ?z }
FILTER(?z = :a)
}
result:
No rows : The OPTIONAL does not match, so the result to that point is one row, not bindings. The FILTER fails, ?z is not bound. Hence result is no rows.
With the patch (the change around line 132 is the critical point):
One row, ?z bound to :a caused by assign to the empty row.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438928#comment-13438928 ]
Andy Seaborne commented on JENA-294:
------------------------------------
This is making it a moving target.
Please provide tests for jena-arq/testing/ARQ/OptFilterEquality/ so there is a fixed target to aim for.
and surely the query is equivalent in outcome to the more conventional:
SELECT *
{
?x :p ?o2
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
}
Optimization has a degree of pragmatism - more conventional cases get better treatment. All queries get the correct results.
)
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne updated JENA-294:
-------------------------------
Issue Type: Improvement (was: Bug)
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434393#comment-13434393 ]
Andy Seaborne commented on JENA-294:
------------------------------------
You suggestion has something in it but pushing the assign into the RHS will need changes in the actual rewrite, not just the pre-condition safeToTransform. It's chaning the shape of the algebra tree in a way the current transformation code does not do (function subst calls the Substitution engine) - this will need to happen in processFilterWorker.
It may not be necessary, or it may be simpler, after leftjoin to condition as the precondition for that rewrite depends on scoping of variables as well.
What we need is comprehensive set of tests.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434381#comment-13434381 ]
Simon Helsen commented on JENA-294:
-----------------------------------
how about changing the check around line 132 to the following:
if (isUnitTable(opLeft)) {
opLeft = opleftjoin.getRight();
}
I guess this is what you suggest in your comment above, except for more general cases than just the unit left hand side.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435985#comment-13435985 ]
Andy Seaborne commented on JENA-294:
------------------------------------
I was agreeing with you - it does not do the right thing. See the tests - filter-equality-08
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: jena-294-afs-1.patch, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434393#comment-13434393 ]
Andy Seaborne edited comment on JENA-294 at 8/15/12 6:11 PM:
-------------------------------------------------------------
Your suggestion has something in it but pushing the assign into the RHS will need changes in the actual rewrite, not just the pre-condition safeToTransform. It's chaning the shape of the algebra tree in a way the current transformation code does not do (function subst calls the Substitution engine) - this will need to happen in processFilterWorker.
It may not be necessary, or it may be simpler, after leftjoin to condition as the precondition for that rewrite depends on scoping of variables as well.
What we need is comprehensive set of tests.
was (Author: andy.seaborne):
You suggestion has something in it but pushing the assign into the RHS will need changes in the actual rewrite, not just the pre-condition safeToTransform. It's chaning the shape of the algebra tree in a way the current transformation code does not do (function subst calls the Substitution engine) - this will need to happen in processFilterWorker.
It may not be necessary, or it may be simpler, after leftjoin to condition as the precondition for that rewrite depends on scoping of variables as well.
What we need is comprehensive set of tests.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Helsen updated JENA-294:
------------------------------
Attachment: patch.txt
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434431#comment-13434431 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Fixing the simple case and only the simple case is in danger of breaking other queries. I guess your internal use case has an optional that, in practice, is matched. That works in various ways.
I tried your suggestion and for
SELECT *
{
OPTIONAL { ?x :q ?o }
FILTER(?x = :x)
}
I get
(assign ((?x :x))
(conditional
(table unit)
(bgp (triple :x :q ?o))))
which is always assigning, even when the leftjoin/conditional does not match. It should remove the row by filter. See full example above.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-294) TransformFilterEquality does not handle
starting OPTIONAL well
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Helsen updated JENA-294:
------------------------------
Attachment: manifestPatchForOptFilterEquality.txt
filter-equality-13.srj
filter-equality-13.rq
for completeness and because you asked.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Fix For: ARQ 2.9.4
>
> Attachments: filter-equality-13.rq, filter-equality-13.srj, jena-294-afs-1.patch, manifestPatchForOptFilterEquality.txt, patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-294) TransformFilterEquality does not
handle starting OPTIONAL well
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435001#comment-13435001 ]
Andy Seaborne commented on JENA-294:
------------------------------------
Tests for correct application of optimization transforms go in TestOptimizer.
> TransformFilterEquality does not handle starting OPTIONAL well
> --------------------------------------------------------------
>
> Key: JENA-294
> URL: https://issues.apache.org/jira/browse/JENA-294
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: Jena 2.7.4
> Reporter: Simon Helsen
> Attachments: patch.txt, patch.txt, test-filter-equality.zip
>
>
> There was one other case where our tests were stuck on a very slow query execution because transformFilterEquality failed to optimize. The problem is that the optimizer gives up whenever the WHERE clause starts with an OPTIONAL clause. The reason is that the generated algebraic formula starts with a TableUnit and this is not handled correctly.
> I have attached a patch which fixes the problem
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira