You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Martin Alderson (JIRA)" <ji...@apache.org> on 2007/05/31 15:25:17 UTC
[jira] Created: (DIRSERVER-951) Negated filter on indexed attribute
doesn't find entries without attribute
Negated filter on indexed attribute doesn't find entries without attribute
--------------------------------------------------------------------------
Key: DIRSERVER-951
URL: https://issues.apache.org/jira/browse/DIRSERVER-951
Project: Directory ApacheDS
Issue Type: Bug
Components: core
Affects Versions: 1.5.1
Reporter: Martin Alderson
Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12520671 ]
Alex Karasulu commented on DIRSERVER-951:
-----------------------------------------
Martin can you comment on this issue as well. I just want to make sure you're not seeing this bug anymore.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-951) Negated filter on indexed attribute
doesn't find entries without attribute
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRSERVER-951:
----------------------------------------
Fix Version/s: 1.5.1
It's important to fix this problem in the current trunk
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Karasulu reassigned DIRSERVER-951:
---------------------------------------
Assignee: Alex Karasulu
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Martin Alderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Alderson resolved DIRSERVER-951.
---------------------------------------
Resolution: Fixed
Committed the previously mentioned fix and enabled the integration test for it.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Martin Alderson
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Martin Alderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500367 ]
Martin Alderson commented on DIRSERVER-951:
-------------------------------------------
I only found the problem by using a more complicated filter (along with being restricted to a specfic base context). The not filter would just require an extra filtering step on the set of results returned by the scope filter (or some other part of the filter which is more optimal). So in my case, this would not be worse than using any other kind of filter.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DIRSERVER-951) Negated filter on indexed attribute
doesn't find entries without attribute
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny closed DIRSERVER-951.
---------------------------------------
closed
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Martin Alderson
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRSERVER-951) Negated filter on indexed attribute
doesn't find entries without attribute
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Karasulu updated DIRSERVER-951:
------------------------------------
Priority: Blocker (was: Major)
Don't know why but many issues with the NOT operator have recently started to appear. I need to take a deep look into what's going on here overall regarding (!) handling. I am increasing this bug up to the blocker level because until it is fixed we should not be releasing.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
> Priority: Blocker
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Martin Alderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Alderson reopened DIRSERVER-951:
---------------------------------------
This is still a problem but it's rarer than I thought.
I have created a new test case based on yours (Alex) to show the problem. It is currently disabled (since it fails!)
TEST: http://svn.apache.org/viewvc/directory/apacheds/trunk/server-unit/src/test/java/org/apache/directory/server/DIRSERVER951ITest.java?view=markup&pathrev=567283
LDIF: http://svn.apache.org/viewvc/directory/apacheds/trunk/server-unit/src/test/resources/org/apache/directory/server/DIRSERVER951ITest.ldif?view=markup&pathrev=567280
Your test wasn't provoking the problem for two reasons:
1. The indexed attribute you used was ou. This attribute type has a subtype (c-ou) and the NormalizationService expands the filter from (!(ou=drama)) to (!(|(2.5.4.11=drama)(2.5.4.11.1=drama))). The problem will only occur when the negated filter is a leaf node (i.e. not an AND, OR, or another negation). This means the problem will only affect indexed attributes that do not have a subtype.
2. The scope part of the filter (base DN of ou=actors...) will be preferred over the negation filter by the filter optimizer. The problem only occurs when the negated filter is selected by the optimizer as the primary filter.
My new test gets round requirement 1 by testing CN instead of OU, and requirement 2 by doing a search on the entire system partition rather than just ou=actors,ou=system.
-
The problem is that the current code is assuming that if the attribute is indexed then we can just ignore all those entries that do not have this attribute. The javadoc for org.apache.directory.server.core.partition.impl.btree.DefaultOptimizer#getNegationScan gives away the current code's intentions:
* Negation counts are estimated in one of two ways depending on its
* composition. If the sole child of the negation is a leaf and an index
* exists for the attribute of the leaf then the count on the index is taken
* as the scan count. If the child is a branch node then the count of the
* negation node is set to the total count of entries in the master table.
* This last resort tactic is used to get a rough estimate because it would
* cost too much to get an exact estimate on the count of a negation on a
* branch node.
I think to fix this problem we need to change the bit of org.apache.directory.server.core.partition.impl.btree.ExpressionEnumerator#enumNeg that gets the base enumeration. We can change:
// Iterates over entire set of index values
if ( node.getChild().isLeaf() )
{
LeafNode child = ( LeafNode ) node.getChild();
if ( db.hasUserIndexOn( child.getAttribute() ) )
{
idx = db.getUserIndex( child.getAttribute() );
childEnumeration = idx.listIndices();
}
else
{
childEnumeration = db.getNdnIndex().listIndices();
}
}
// Iterates over the entire set of entries
else
{
idx = db.getNdnIndex();
childEnumeration = idx.listIndices();
}
..to...
childEnumeration = db.getNdnIndex().listIndices();
We perhaps should also change the name of childEnumeration to baseEnumeration - we were never actually getting an enumeration for the child filter.
The old code seems to be optimizing based on an incorrect assumption. I would like someone to agree with me that this is the right move before I go ahead and commit this change though. Assign the issue to me if you want me to take care of it.
In addition, if this fix is made we should also change org.apache.directory.server.core.partition.impl.btree.DefaultOptimizer#getNegationScan to just return MAX - a negation scan is always worst case since we have to retrieve all entries from the partition then iterate over them filtering out those that do not pass the negated test. In the other filter optimizers where we can't use an index we just return MAX too. Of course, I think this will probably make it impossible to meet requirement 2 since a scope filter node will always be preferred over the negation filter, unless the filter optimizer is disabled.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500365 ]
Emmanuel Lecharny commented on DIRSERVER-951:
---------------------------------------------
If so, then this is clearly a bug.
However, for performance reasons, I suggest you to avoid using *not* filters, even on indexed attributes. Doing so will simply result on a full scan, with rejection of entries which attribute exist. Of course, this is just and advice, and sometime, it's simply not possible ...
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DIRSERVER-951) Negated filter on indexed
attribute doesn't find entries without attribute
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Karasulu reassigned DIRSERVER-951:
---------------------------------------
Assignee: Martin Alderson (was: Alex Karasulu)
Martin this is a very inciteful synopsis of the problem. I'm amazed by your understanding of the server. I have not assessed your reasoning fully yet and I will as well as keep an eye on the commits. I'm turning over this issue to you. I'll be offline today to take care of a few things but I'll be available later if you want to talk about it.
Also please make sure you run all integration tests before committing your changes just to make sure all is legit. I will reopen this issue as well.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Martin Alderson
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DIRSERVER-951) Negated filter on indexed attribute
doesn't find entries without attribute
Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Karasulu closed DIRSERVER-951.
-----------------------------------
Resolution: Cannot Reproduce
I created a special integration test just for testing NOT operators in filters with the server with both indexed and non-indexed attributes using with the NOT operator. Here's the test case for this:
http://svn.apache.org/viewvc/directory/apacheds/trunk/server-unit/src/test/java/org/apache/directory/server/NegationOperatorITest.java?revision=566925&view=markup
For some reason I could not reproduce the bug which Martin you sited. If you take a look at the following tests which pass you'll see that (!(ou=drama)) successfully matches for Jack Black and Joe Newbie. According to this LDIF here:
http://svn.apache.org/viewvc/directory/apacheds/trunk/server-unit/src/test/resources/org/apache/directory/server/NegationOperatorITest.ldif?revision=566925&view=markup
Test to consider:
-----------------------
testSearchNotDrama()
testSearchNotDramaIndexed() [SHOULD FAIL]
Both succeed and Jack Black whose entry is returned does have the ou attribute but no 'drama' value. Also Joe Newbie is returned as well and this entry has no ou attribute all together.
So unless this can be reproduced some other way (feel free to add a test) then I'm going to just mark this issue closed and not reproducable.
> Negated filter on indexed attribute doesn't find entries without attribute
> --------------------------------------------------------------------------
>
> Key: DIRSERVER-951
> URL: https://issues.apache.org/jira/browse/DIRSERVER-951
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.5.1
> Reporter: Martin Alderson
> Assignee: Alex Karasulu
> Priority: Blocker
> Fix For: 1.5.1
>
>
> Searching with filter (!(myAttribute=value)) will not find entries which do not have a myAttribute attribute when that attribute is indexed. When myAttribute is not indexed the filter works as expected, finding all entries that either do not have the specified value for myAttribute or do not have any values for myAttribute at all.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.