You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Emmanuel Lecharny (JIRA)" <ji...@apache.org> on 2009/06/18 12:15:07 UTC
[jira] Created: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Potential concurrency issue when adding/modifying/deleting entries at a high rate
---------------------------------------------------------------------------------
Key: DIRSERVER-1377
URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
Project: Directory ApacheDS
Issue Type: Bug
Affects Versions: 1.5.4
Reporter: Emmanuel Lecharny
Priority: Blocker
Fix For: 1.5.5
When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Ioannis Mavroukakis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721171#action_12721171 ]
Ioannis Mavroukakis commented on DIRSERVER-1377:
------------------------------------------------
Have you thought about using a CountDown latch? I've found them to be extremely useful as gatekeepers when I am waiting on something else to happen before I allow a concurrent method to execute.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Bharat Gera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729584#action_12729584 ]
Bharat Gera commented on DIRSERVER-1377:
----------------------------------------
The developers are busy at the moment. May have a test result next week.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721720#action_12721720 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
After a bit of code browsing, it's not exactly simple to fix.
An index contains tuple<Key, value>. The value can be unique or we can have more than one, and in this case, it will be stored into a AvlTree<value> or BTree<value>.
The Key can be anything in :
- Long
- StringValue
- BinaryValue
- DN
- RDN (soon)
The value can be anything, in two categories.
* Simple types :
- Long
- StringValue
- BinaryValue
- DN
- RDN (soon)
* Composite types
- AvlTree<Simpletype>
- BTree<SimpleType>
We have to design a de/ser which handle all those cases.
The revert index is working in the opposite way : tuple<Value, Key> but in fact, if you forget about what is the key, and what is the value, it's exactly the same as a forward index : the Value will be the key and the kay the value (and it can also be a composite type.
For instance, if we consider the ObjectClass index, we have :
forward :
<'top', [1,2,3]>
<'person', [1,2]>
<'inetOrgPerson', 1>
<'organizationalUnit', 3>
reverse :
<1, ['top', 'person', 'inetOrgPerson']>
<2, ['top', 'person']>
<3, ['top', 'organizationalUnit']>
All the serialization and deserialization *must* be done in JDBM, not before, otherwise we might have some inconsistency.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Bharat Gera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728786#action_12728786 ]
Bharat Gera commented on DIRSERVER-1377:
----------------------------------------
That is really good. I'm Ok even if we have a solution where the risk of the bug is mitigated to a large extent. Solving it completely is ideal though.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Resolved: (DIRSERVER-1377) Potential concurrency issue
when adding/modifying/deleting entries at a high rate
Posted by Emmanuel Lecharny <el...@apache.org>.
Alex Karasulu wrote:
> Can we have a solid summary of this case with the nature of the fix
> somewhere in case the problem comes back ? From what I hear it was pretty
> damn hairy.
>
I think that all the summary can be found in DIRSERVER-1377 and 1359, in
the comments.
I will try to summarize the fix and the way to the fix soon.
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: [jira] Resolved: (DIRSERVER-1377) Potential concurrency issue
when adding/modifying/deleting entries at a high rate
Posted by Alex Karasulu <ak...@gmail.com>.
Can we have a solid summary of this case with the nature of the fix
somewhere in case the problem comes back ? From what I hear it was pretty
damn hairy.
Thanks guys, a great job.
Regards,
Alex
On Sat, Jul 18, 2009 at 12:28 AM, Kiran Ayyagari <ay...@gmail.com>wrote:
>
> Thanks Emmanuel for the great effort you put in fixing this, I can't forget
> the fact about the amount of logs you generated and grepped through (10GB?)
>
> Kiran Ayyagari
>
>
> [
>> https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>>
>> Emmanuel Lecharny resolved DIRSERVER-1377.
>> ------------------------------------------
>>
>> Resolution: Fixed
>>
>> Should be fixed with commits :
>> http://svn.apache.org/viewvc?rev=795285&view=rev
>> http://svn.apache.org/viewvc?rev=795286&view=rev
>> http://svn.apache.org/viewvc?rev=795287&view=rev
>> http://svn.apache.org/viewvc?rev=795289&view=rev
>>
>> If not, please reopen the issue.
>>
>>
>
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
Re: [jira] Resolved: (DIRSERVER-1377) Potential concurrency issue
when adding/modifying/deleting entries at a high rate
Posted by Kiran Ayyagari <ay...@gmail.com>.
Thanks Emmanuel for the great effort you put in fixing this, I can't forget
the fact about the amount of logs you generated and grepped through (10GB?)
Kiran Ayyagari
> [ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Emmanuel Lecharny resolved DIRSERVER-1377.
> ------------------------------------------
>
> Resolution: Fixed
>
> Should be fixed with commits :
> http://svn.apache.org/viewvc?rev=795285&view=rev
> http://svn.apache.org/viewvc?rev=795286&view=rev
> http://svn.apache.org/viewvc?rev=795287&view=rev
> http://svn.apache.org/viewvc?rev=795289&view=rev
>
> If not, please reopen the issue.
>
[jira] Resolved: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny resolved DIRSERVER-1377.
------------------------------------------
Resolution: Fixed
Should be fixed with commits :
http://svn.apache.org/viewvc?rev=795285&view=rev
http://svn.apache.org/viewvc?rev=795286&view=rev
http://svn.apache.org/viewvc?rev=795287&view=rev
http://svn.apache.org/viewvc?rev=795289&view=rev
If not, please reopen the issue.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729216#action_12729216 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
I have committed all what we were working on this morning :
http://svn.apache.org/viewvc?rev=792502&view=rev
Can you give it a try and tell us if it works fine now ?
(We have other patches to inject before we close this issue anyway)
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721164#action_12721164 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
I have added some traces, we get a NPE in the AvlTreeMarshaller (the tree dump at the end) :
java.lang.NullPointerException
at org.apache.directory.server.core.avltree.AvlTreeMarshaller.deserialize(AvlTreeMarshaller.java:210)
at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.getDupsContainer(JdbmTable.java:864)
at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.put(JdbmTable.java:593)
at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.add(JdbmIndex.java:458)
at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmStore.add(JdbmStore.java:1221)
at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition.add(JdbmPartition.java:584)
at org.apache.directory.server.core.partition.DefaultPartitionNexus.add(DefaultPartitionNexus.java:849)
at org.apache.directory.server.core.interceptor.InterceptorChain$1.add(InterceptorChain.java:137)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.trigger.TriggerInterceptor.add(TriggerInterceptor.java:282)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.event.EventInterceptor.add(EventInterceptor.java:153)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.collective.CollectiveAttributeInterceptor.add(CollectiveAttributeInterceptor.java:352)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.subtree.SubentryInterceptor.add(SubentryInterceptor.java:582)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.schema.SchemaInterceptor.add(SchemaInterceptor.java:1872)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.operational.OperationalAttributeInterceptor.add(OperationalAttributeInterceptor.java:202)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.exception.ExceptionInterceptor.add(ExceptionInterceptor.java:220)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.interceptor.BaseInterceptor.add(BaseInterceptor.java:129)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.authz.AciAuthorizationInterceptor.add(AciAuthorizationInterceptor.java:448)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.referral.ReferralInterceptor.add(ReferralInterceptor.java:251)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.authn.AuthenticationInterceptor.add(AuthenticationInterceptor.java:212)
at org.apache.directory.server.core.interceptor.InterceptorChain$Entry$1.add(InterceptorChain.java:1196)
at org.apache.directory.server.core.normalization.NormalizationInterceptor.add(NormalizationInterceptor.java:117)
at org.apache.directory.server.core.interceptor.InterceptorChain.add(InterceptorChain.java:756)
at org.apache.directory.server.core.DefaultOperationManager.add(DefaultOperationManager.java:260)
at org.apache.directory.server.core.DefaultCoreSession.add(DefaultCoreSession.java:183)
at org.apache.directory.server.core.DefaultCoreSession.add(DefaultCoreSession.java:169)
at org.apache.directory.server.ldap.handlers.AddHandler.handle(AddHandler.java:57)
at org.apache.directory.server.ldap.handlers.AddHandler.handle(AddHandler.java:39)
at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:176)
at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:1)
at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:194)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:721)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:433)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:801)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:71)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:801)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:793)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:735)
at java.lang.Thread.run(Thread.java:619)
Bad tree : [0x00 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x67 0x00 0x00 0x00 0x0A 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x07 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x06 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x07 0x00 0x00 0x00 0x06 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x0A 0x00 0x00 0x00 0x09 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x09 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x73 0x00 0x00 0x00 0x11 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x6F 0x00 0x00 0x00 0x0D 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x6D 0x00 0x00 0x00 0x0B 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x71 0x00 0x00 0x00 0x0F 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x70 0x00 0x00 0x00 0x0E 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x72 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x74 0x00 0x00 0x00 0x12 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x6C 0x78 0x00 0x00 0x00 0x13 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ]
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny closed DIRSERVER-1377.
----------------------------------------
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732609#action_12732609 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
I have substituted the current AvlTree we were using to store duplicates in index by a simpler array of Objects, as I suspect a bug in the way we handle AvlTrees in very specific cases to be the cause of the problem we have.
So far, tests seems to be ok, as I was able to inject more than 1 500 000 entries in the server without problem (20 concurrent threads creating and deleting entries concurrently).
I will probably commit the change tonite.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722581#action_12722581 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
After some more investigations, the problem is narrowed to the JdbmTable.put(K, V) operation, not necessarily on the oneLevelIndex (but most often on this index).
A possible bug in JDBM ?
At this point, I see no other option than to remove all the surrounding code and do a direct tester on the index. It will be faster to determinate what can be the problem...
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728694#action_12728694 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
After days of painfull debug sessions, code analysis and log grepping, we have moved the serialization in the backend (JDBM), synchronized correctly JdbmStore, and it seems to have a positive impact.
I've run some test, and was able to inject more than 1 400 000 updates without any problem.
But the oneLevelIndex is still corrupted, we have to understand why.
The idea is to replace the AVLTree by a simple array, which will be sorted. It should be a bit less expensive in CPU, will suck less memory, and cost more for updates.
This is what I'm on right now.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728796#action_12728796 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
As of today, here is the status :
- I have run 2 000 000 add/delete using the MultiThreadedTest class. All was fine
- *but* the OneLevel index is broken. We can't any more do searches using this index, after the test has run
- more precisely, the index is corrupted after only a few thousands of updates
- I can't craft a scenario corrupting the index, so it's pretty hard to understand why it happens.
We decided to commit the current code, because in any case, it's way safer than the code we have. We also want to substitute the AVLTree we are using internally with a simple array, allowing us to have a better control, and potentially less odd that it can be a bug in AvlTree
Another action would be to test the server with this MultiThreadedTest against a previous version of the server (1.5.3) to see if this issue was present then.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722181#action_12722181 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
Index operations must be ACID : when modifying the values, no other thread should be able to access them until the modification is done. That mean we must provide a copy of the values when a search is done, as we can't keep a lock on the associated tree while the server is generating the response (it might take way too long).
Each index must be protected against concurrent access.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722995#action_12722995 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
Update :
After having put some more logs, and grepped/sedded gigabytes of logs (no fun at all), I have found some dubious code and result. Here is the log I obtain just before the first NPE :
---> pool-6-thread-7 - ---> Remove apacheOneLevel_forward = 1, 226034
pool-6-thread-7 - <--- Remove AVL apacheOneLevel_forward = 1, 226034
pool-6-thread-7 - ---> Remove apacheOneLevel_reverse = 226034
pool-6-thread-7 - <--- Remove AVL apacheOneLevel_reverse = 226034
pool-6-thread-7 - ---> Remove apacheSubLevel_forward = 226034, 226034
pool-6-thread-7 - <--- Remove AVL apacheSubLevel_forward = 226034, 226034
pool-6-thread-7 - ---> Remove apacheSubLevel_reverse = 226034
pool-6-thread-7 - <--- Remove AVL apacheSubLevel_reverse = 226034
---> pool-6-thread-7 - ---> Remove apacheOneLevel_forward = 1, 226034
pool-6-thread-7 - Error while removing 1, 226034 on table apacheOneLevel_forward
java.lang.NullPointerException
at org.apache.directory.server.core.avltree.AvlTreeMarshaller.deserialize(AvlTreeMarshaller.java:240)
As one can see, we remove twice something from the oneLevelIndex. The code which does that is :
ndnIdx.drop( id );
updnIdx.drop( id );
* oneLevelIdx.drop( id );
entryCsnIdx.drop( id );
entryUuidIdx.drop( id );
if( id != 1 )
{
subLevelIdx.drop( id );
}
// Remove parent's reference to entry only if entry is not the upSuffix
if ( !parentId.equals( 0L ) )
{
* oneLevelIdx.drop( parentId, id );
}
Sadly (?) the second removal has no impact (I checked it), except sucking CPU.
Otherwise, *all* the exceptions I get are from the same line of code : AvlTreeMarshaller.java:240
for( int i = 0; i < nodes.length - 1; i++ )
{
nodes[ i ].setNext( nodes[ i + 1] );
* nodes[ i + 1].setPrevious( nodes[ i ] );
}
which means the index table is broken (nodes[i+1] is null). There are two possibilities for this table to be incorrect :
- we save some bad table
- we recreate a bad table
I will double check that code.
Otherwise, we discussed a lot with Kiran about serialization/deserialization, and we agreed that it should be done *inside* jdbm. Will work on that too.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721820#action_12721820 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
Well, you still have to handle all the cases, as a JdbmTable not allowing duplicates will have a simple type *only* and the JdbmTable allowing duplicates will allow *only* composite types. It just split the problem in two parts.
But IMO, you have to deal with simple type.
--- AvlTree manipulation ---
When you have deserialized the AvlTree, you have a data structure which is totally thread safe. The problem is when you deserialize it from a byte[] which is a reference to something that can be modified by another thread, as it's a reference. If you do the de/ser into JDBM, as the retrieve/remove/put methos are synchronized, this is safe, as no thread can touch the byte[] while you are deserializing it.
Now, it will be costly. Another optuion would be to have a ByteArraySerializer which just do a copy of the internal byte[], and returns this copy to the user. The thread will then deserialize something that it owns. Same thing for serialization, except that you don't need to copy the byte[], as it's produced from scratch when serializing the tree.
IMO, this is probably the way to implement the fix : easy (no need to rewrite all the existing marshallers) and will be faster than what we have in mind atm.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721802#action_12721802 ]
Kiran Ayyagari commented on DIRSERVER-1377:
-------------------------------------------
Though theoretically all the above stands, implementation of the de/serializer under question is a bit simple because
The JdbmTable (on which JdbmIndex is based) does an internal check about allowing duplicate keys, *iff* duplicates are allowed then
it creates an AvlTree and stores the value(s) in it. Note that even if a key has a single value and the duplicates are allowed in a table
then the value will be stored in a AvlTree.
If the size of this AvlTree increases beyond a threshold then this AvlTree values will be copied to a Jdbm BTree and the record Id is stored instead of
the AvlTree.
So the only values that need to be taken care by the serializer are 1. AvlTree or 2. BTreeRedirect (which holds the record Id of jdbm BTree)
--- Update about fix --
Have tried with the new serializer but that didn't solve the problem, IMO the issue is with the way the AvlTree is modified in the JdbmTable's put() and remove() methods.
To test this I have synchronized these methods and haven't encountered the NPE. This may not be an acceptable solution, but just to mention.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721289#action_12721289 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
Ok, it seems that the problem comes from the way we manage AVLTree and store them in the JDBM backend : serialization and deserialization are done *before* the data are stored or retreived from JDBM. JDBM stores and return a reference to an Object, and in this case, it will be a byte[]. Obvioulsy, this reference can be modified while we are deserializing it if another thread is modifying it.
We just have to declare the de/serializer to be JDBM de/serializer, the way we did for ServerEntry, and attach those de/serializer to the JDBMTable.
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRSERVER-1377) Potential concurrency issue when
adding/modifying/deleting entries at a high rate
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721174#action_12721174 ]
Emmanuel Lecharny commented on DIRSERVER-1377:
----------------------------------------------
This is not as easy as it seems. If we were to use some Java 5 concurrency object, I think it would be a ReadWriteLock. Now, the problem is that the index are used by the search operations, so we can't brutally synchronize the whole indexes while doing a write, as searches can last forever (for instance, when dealing with PersistentSearch, for replication).
We must provide a protection against concurrent acces by guaranteeing that the index are safe, not their content (ie, the elements contained in the index may have changed since the search has started, but the next() operation should always return something safe).
This is not simple...
> Potential concurrency issue when adding/modifying/deleting entries at a high rate
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1377
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1377
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.4
> Reporter: Emmanuel Lecharny
> Priority: Blocker
> Fix For: 1.5.5
>
>
> When adding/deleting entries with many clients (a client add and delete an entry many times), we may have some concurrency problem, as the index are updated without concurrent acces protection.
> Synchronizing the classes where we update the index might help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.