Re: Txn Work Update(2)

[Selcuk AYA <ay...@gmail.com>]

On Wed, Nov 30, 2011 at 3:26 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/30/11 11:48 AM, Selcuk AYA wrote:
>>
>> Hi all,
>
>
> hi !
>
>> time to give an update on the txn status. I just checked in some code
>> with explanation of the changes for each project separately. Here is
>> just an overall summary and todo:
>>
>> *I added txn being/commit/abort to ldap requests and ran some test
>> with txns enabled. Put in various fixes to the txn layer based on the
>> tests. So first signs life signs for txns. More tests will be done but
>> I need to digress into AVL first. I also need to handle paged search.
>
> Great !
>
> Paged search is just a a normal search, but one that last longer. The way we
> handle it is by using a counter of sent responses.
>
> Basically, entries are sent one by one to the client, using a
> SearchResultEntry to hold the entry. With the pagedSearch, once we have
> reached the limit set by the client, we stop the sending of entries, and
> hold the cursor in the user's session, so that he can send back a new search
> request for then next entries - if needed -. Is there anything special you
> need to deal with here ? (btw, the cursor will be deleted as soon as the
> session will be closed. I hope this cleanup is effective...)
>
There is not much todo here as you guessed. The only problem is the
transaction context is stored in a thread local variable. When paged
search send partial data, we should store the context somewhere and
hand it off to the thread that will continue processing the search.

>
>>
>> *I have also been working on removing interfaces related to
>> modification from the partition
>
> Hmmm, can you be a bit more specific ?

When I say modification or change, I mean
modify/move/rename/moveAndRename/add/delete. Instead of partitions
implementing these separately, there is a layer above partitions which
handle these operations transactionally. The class that handles this
is called OperationExecutionManager. Partitions expose master table
and indices for the operation execution manager execute the
operations.

>
>
>> and have been trying to get rid of the
>> Store interface. Got rid of the store interface except for config
>> methods and calls from the ui layer. I removed the modifications up
>> but still have to call the modification methods of the partition
>> interface because SchemaPartition triggers some changes based on the
>> logical  change. I believe those changes can be removed to schema
>> interceptor?
>
> We can't modify a schema. We can only add or delete a schema entry. This is
> a limitation, and I think it's mandated by the spec (don't remember where it
> comes from, but the idea is that modifying a schema element might imply a
> global modification on all the entries, something we don't want to accept).
>
>
>> In general, if a partition wants to do a change based on
>> a data update, then it should be able to do it during master table
>> update or by registering an interceptor(For example I changed multi
>> file ldif partition to do file updates when master table is updated).
>> Then we can get rid of the modification methods on partitions.
>
> you mean, the modify/move/rename/moveAndRename methods ?

As mentioned above, by change or modification, I mean any operation
that ends up in update of data. For schema partitions, I believe
changes triggered after logical ldap change operations can be moved to
an interceptor.

>
>>
>> Note that, how search is done is still upto the partiton.
>> DefaultSearchEngine is changed to work with any partition and returns
>> transactioanally consistent data.( it used to work with Store
>> interface. Now it works with Partition interface. ). However, a
>> partition can use another search engine and can use the data in txn
>> logs to provide consistent data.
>
>
> Ok.
>
>>
>> * Would be good to get triggers working as they are a nice way to
>> demonstrate txns. What is the reason trigger tests are ignored?
>
> Because since we have modified the server to not use JNDI anymore, it
> impacted the trigger code (which is still based on JNDI). We don't have time
> now to fix this feature, and we have to review the full code in this area,
> as the guy who contributed it is now gone and missing. Let's say it was a
> choice : we know that it's a cool feature, but it's not an urgent one atm...
>
>>
>> * Would be good to have statement snapshot isolation for
>> EntryFilteringCursor. Not having this doesnt affect ldap requests but
>> might affect code that works with EntryFilteringCursor directly.
>> Currently there are few cases of this especially during startup.
>
>
> I'll buy some more explanation : what is a 'statement snapshot' ?

Say you do a search and get an entry filtering cursor and then you
modify the entries that you get from the cursor. Something like this:

beginTransaction()
cursor = ..
while ( cursor.next() )
{
  entry = cursor.get();
  modify entry.
}

endTransaction()

If we have statement snapshot for the search, cursor will not see any
updates to the master table of index tables done through the
modifications.  This might be useful in some cases.

>
>>
>> * Tried to introduce some rules into how modifications are done: If we
>> think of search engine and data modification layer as a data layer,
>> then under it we have storage level. Storage level might have a cache
>> of entries, or index entries or tuples. When a thread reads an object
>> from the storage layer and wants to update it, another thread might be
>> reading the same object. To avoid confusing the reader thread, object
>> to be changed should be cloned before change is applied.
>
> Absolutely.
>
>> Changed
>> modification layer to follow this rule. As mentioned in the previous
>> emails, default search engine is modifying index entries and this is
>> not good, I  put a workaround for this.
>
> Ok. We need to fix this.
>
> Thanks a lot for the update, Selcuk !
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>

Re: Txn Work Update(2)

[Emmanuel Lecharny <el...@gmail.com>]

On 12/15/11 5:04 PM, Selcuk AYA wrote:
> On Thu, Dec 15, 2011 at 2:27 PM, Emmanuel Lecharny<el...@gmail.com>  wrote:
>> On 12/15/11 12:25 PM, Selcuk AYA wrote:
>>> Hi all,
>>>
>>> update on my status:
>>>
>>> * There is going to be a big change in my life soon and I will do a
>>> big move. So I will be MIA for some time( around 2 months) and after
>>> that I will have less time to contribute but I will try to keep being
>>> involved in development and discussions.
>> Ok. I wish this is a move for a better life :)
>>
>>
>>> * I did a recent check in. There are some tests that fail. I will list
>>> them below. The basic reason that those tests are failing is that they
>>> do queries and updates of which  some are transactional, some are not
>>> and this leads to wrong results.
>> I spent some time yesterday on the failing tests. What I need here is a clue
>> about how to make some operation transactionals. If you can update us, that
>> would be great.
> please take a look at ldap protocol handlers.

Ok, got it. It's about doing :

txnManager.beginTransaction( false );
...
txnManager.commitTransaction();

(or txnManager.abortTransaction() if we've got an error)

When should we put a 'true' as a parameter ?

>>
>>> And the basic reason they do queries
>>> and updates in a non transactional way is that I did not put
>>> transaction begin/commif/abort( ie, txn demarcation) at CoreSession.
>>> This is the layer we start the execution of  the basic ldap operations
>>> but I thought there might be clients of CoreSession which want to
>>> group operations together and execute them as a single txn(If you
>>> think this is not necessary, please let me know).
>> I don't think it's anymore necessary. We have refactored all the operations
>> last month so that we never have to re-enter the chain twice when processing
>> an operation.
> I was thinking of clients of CoreSession within ApacheDs. For example,
> revert of changelog or changepassword.

I see. Actually, we get errors when reverting some modification 
operations, so I guess we must add some txns boundaries in the 
processing of reverts.

> There might be other cases.  I
> am not talking about the interceptor chain here. If these kind of
> clients do not need to group ldap operations together as a
> transaction, then move txn begin/commit/abort to CoreSession, and for
> search close the transaction when entry filtering cursor is closed.
>
> Note that, there is also the case that all over the server, cursors
> are not closed or not closed in a finally clause. I think these should
> be fixed.

Definitively something we have to fix. Reminds me the JNDI 
NamingEnumeration that requires to be closed explicitely :/
>
>>
>>>   Also, there is at
>>> least search, for which the cursor is kept open above CoreSession, So
>>> for search, the caller has to handle txn commit, abort.Also there
>>> might be some cases where operations do not use session at all.
>> Well, we do some lookups directly on the nexus, as we are inside the chain
>> already, and I guess we are using the ongping transaction here. If it's not
>> the case, then we should probably have a way to make those operations part
>> of the ongoing txn (don't know how difficult it is)
> Operations within interceptor chain is part of the same transaction,
> But we have some cases which do not use sessions and which directly
> interact with nexus outside interceptor chain. For example some
> interceptor initializations.
Ok.

>>
>>>
>>> As a result, I decided to handle each  caller of CoreSession
>>> operations separately and this increased the number changes that had
>>> to be done. I handled ldap protocol handlers and change password.
>>> Right now StockServerSuite is passing except three tests.
>>> StockCoreSuite and MigratedStockCoreSuite is failing. The reason is
>>> they use LDAPCoreConnection and this is not transactional yet.
>> I have some failures in core-integ. Do you have them too ?
> StockCoreSuite and MigratedStockCoreSuite  =>  Yes . These are using
> LdapCoreConnection. If you move txn begin/end to CoreSession you wont
> need to handle LDAPCoreConnection separately.

Ok.


Thanks for the heads up !


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Txn Work Update(2)

[Selcuk AYA <ay...@gmail.com>]

On Thu, Dec 15, 2011 at 2:27 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 12/15/11 12:25 PM, Selcuk AYA wrote:
>>
>> Hi all,
>>
>> update on my status:
>>
>> * There is going to be a big change in my life soon and I will do a
>> big move. So I will be MIA for some time( around 2 months) and after
>> that I will have less time to contribute but I will try to keep being
>> involved in development and discussions.
>
> Ok. I wish this is a move for a better life :)
>
>
>>
>> * I did a recent check in. There are some tests that fail. I will list
>> them below. The basic reason that those tests are failing is that they
>> do queries and updates of which  some are transactional, some are not
>> and this leads to wrong results.
>
> I spent some time yesterday on the failing tests. What I need here is a clue
> about how to make some operation transactionals. If you can update us, that
> would be great.

please take a look at ldap protocol handlers.
>
>
>> And the basic reason they do queries
>> and updates in a non transactional way is that I did not put
>> transaction begin/commif/abort( ie, txn demarcation) at CoreSession.
>> This is the layer we start the execution of  the basic ldap operations
>> but I thought there might be clients of CoreSession which want to
>> group operations together and execute them as a single txn(If you
>> think this is not necessary, please let me know).
>
> I don't think it's anymore necessary. We have refactored all the operations
> last month so that we never have to re-enter the chain twice when processing
> an operation.

I was thinking of clients of CoreSession within ApacheDs. For example,
revert of changelog or changepassword. There might be other cases.  I
am not talking about the interceptor chain here. If these kind of
clients do not need to group ldap operations together as a
transaction, then move txn begin/commit/abort to CoreSession, and for
search close the transaction when entry filtering cursor is closed.

Note that, there is also the case that all over the server, cursors
are not closed or not closed in a finally clause. I think these should
be fixed.

>
>
>>  Also, there is at
>> least search, for which the cursor is kept open above CoreSession, So
>> for search, the caller has to handle txn commit, abort.Also there
>> might be some cases where operations do not use session at all.
>
> Well, we do some lookups directly on the nexus, as we are inside the chain
> already, and I guess we are using the ongping transaction here. If it's not
> the case, then we should probably have a way to make those operations part
> of the ongoing txn (don't know how difficult it is)

Operations within interceptor chain is part of the same transaction,
But we have some cases which do not use sessions and which directly
interact with nexus outside interceptor chain. For example some
interceptor initializations.
>
>
>>
>>
>> As a result, I decided to handle each  caller of CoreSession
>> operations separately and this increased the number changes that had
>> to be done. I handled ldap protocol handlers and change password.
>> Right now StockServerSuite is passing except three tests.
>> StockCoreSuite and MigratedStockCoreSuite is failing. The reason is
>> they use LDAPCoreConnection and this is not transactional yet.
>
> I have some failures in core-integ. Do you have them too ?

StockCoreSuite and MigratedStockCoreSuite  => Yes . These are using
LdapCoreConnection. If you move txn begin/end to CoreSession you wont
need to handle LDAPCoreConnection separately.

>
>>
>> For anybody who wants to get involved, here is the list of todo:
>> * Figure out why SaslBind and DelegatedAuthIT in stockserver suite is
>> failing. We get a KRBException here and I do not know this module well
>> enough.
>
> Will check.
>
>>
>> *Handle LdapCoreConnection. This is used in applications which embed
>> apacheds i guess?
>
> Yes.
>
>> The basic difficulty is that users of this class get
>> a cursor and the transaction should be comitted or aborted only when
>> this cursor closes . Moreover, when a cursor is open, user of
>> ldapcoreconnection can call other ldap operations and it might make
>> sense to group these in a transaction. If this is true, then it should
>> be the responsibility of the callers of LdapCoreSession to start and
>> end txns. If we are not going to let the user do this, we need to
>> store a pointer to txn context in the cursor and end the txn when
>> cursor is closed taking into account that the txn might have to be
>> upgraded to a readwrite txn from a read only txn if the user does
>> changess while holding the cursor open.
>
> I don't think we should expose the txns. It should remain an internal
> mechanism. Also I don't mind if a user has an issue because he is doing some
> changes in the server while processing a search : it's his problem, not our.
> It is very clear that LDAP is not transactionnal, and the suer should be
> aware of this fact. (that just means a user can't expect that a long search
> can return entries which have been modified or deleted since the search has
> started. But the user should not get an exception just because an entry does
> not exists anymore when the cursor is moving forward).
>
> At least, this is my undertsanding. Alex, do you agree on that ?
>
>>
>> * There are some tests which call session operations directly, these
>> should be wrapped in txn begin/end.
>
> ok.

See above comment about moving txn beging/commit/end to CoreSession.

>
>>
>> *I put some tests to AVL and jdbm partitions to test txns without the
>> effect of interceptors and logical caches. These are multithreaded and
>> are good to test the robustness of the data change and query layer and
>> txn subsystem. Add more tests like this.
>
> Ok.
>
>>
>> * Handle logical caches. As we talked handle these with read write
>> locks taking cleanup in  case of an aborted txn into account.
>>
>> Please let me know if anyone wants to attack any of these. I might
>> provide more input if needed.
>
>
> I will spend some time by the end of this year on those things. We need to
> get a branch that builds and passes tests with flying colors, even if it's
> not perfect.
>
> Also I see that there are TODO into the DefaultOperationExecutionManager
> class :
>
>    public void delete( Partition partition, DeleteOperationContext
> deleteContext ) throws LdapException
>    {
>        Dn dn = deleteContext.getDn();
>
>        //TODO TODO TODO REMOVE THIS
>        partition.delete( deleteContext );
>
> What should we do ?
I mentioned previously that I was trying to remove
add/delete/modify/moveDn interfaces from partitions. This reason I
couldnt do that was SchemaPartition was triggning some updates with
these mehtods. Since you mentioned those are moved, these calls can be
removed and partition interface can be changed.

>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>

Re: Txn Work Update(2)

[Emmanuel Lecharny <el...@gmail.com>]

On 12/15/11 12:25 PM, Selcuk AYA wrote:
> Hi all,
>
> update on my status:
>
> * There is going to be a big change in my life soon and I will do a
> big move. So I will be MIA for some time( around 2 months) and after
> that I will have less time to contribute but I will try to keep being
> involved in development and discussions.
Ok. I wish this is a move for a better life :)

>
> * I did a recent check in. There are some tests that fail. I will list
> them below. The basic reason that those tests are failing is that they
> do queries and updates of which  some are transactional, some are not
> and this leads to wrong results.
I spent some time yesterday on the failing tests. What I need here is a 
clue about how to make some operation transactionals. If you can update 
us, that would be great.

> And the basic reason they do queries
> and updates in a non transactional way is that I did not put
> transaction begin/commif/abort( ie, txn demarcation) at CoreSession.
> This is the layer we start the execution of  the basic ldap operations
> but I thought there might be clients of CoreSession which want to
> group operations together and execute them as a single txn(If you
> think this is not necessary, please let me know).
I don't think it's anymore necessary. We have refactored all the 
operations last month so that we never have to re-enter the chain twice 
when processing an operation.

>   Also, there is at
> least search, for which the cursor is kept open above CoreSession, So
> for search, the caller has to handle txn commit, abort.Also there
> might be some cases where operations do not use session at all.
Well, we do some lookups directly on the nexus, as we are inside the 
chain already, and I guess we are using the ongping transaction here. If 
it's not the case, then we should probably have a way to make those 
operations part of the ongoing txn (don't know how difficult it is)

>
>
> As a result, I decided to handle each  caller of CoreSession
> operations separately and this increased the number changes that had
> to be done. I handled ldap protocol handlers and change password.
> Right now StockServerSuite is passing except three tests.
> StockCoreSuite and MigratedStockCoreSuite is failing. The reason is
> they use LDAPCoreConnection and this is not transactional yet.
I have some failures in core-integ. Do you have them too ?
>
> For anybody who wants to get involved, here is the list of todo:
> * Figure out why SaslBind and DelegatedAuthIT in stockserver suite is
> failing. We get a KRBException here and I do not know this module well
> enough.
Will check.
>
> *Handle LdapCoreConnection. This is used in applications which embed
> apacheds i guess?
Yes.
> The basic difficulty is that users of this class get
> a cursor and the transaction should be comitted or aborted only when
> this cursor closes . Moreover, when a cursor is open, user of
> ldapcoreconnection can call other ldap operations and it might make
> sense to group these in a transaction. If this is true, then it should
> be the responsibility of the callers of LdapCoreSession to start and
> end txns. If we are not going to let the user do this, we need to
> store a pointer to txn context in the cursor and end the txn when
> cursor is closed taking into account that the txn might have to be
> upgraded to a readwrite txn from a read only txn if the user does
> changess while holding the cursor open.
I don't think we should expose the txns. It should remain an internal 
mechanism. Also I don't mind if a user has an issue because he is doing 
some changes in the server while processing a search : it's his problem, 
not our. It is very clear that LDAP is not transactionnal, and the suer 
should be aware of this fact. (that just means a user can't expect that 
a long search can return entries which have been modified or deleted 
since the search has started. But the user should not get an exception 
just because an entry does not exists anymore when the cursor is moving 
forward).

At least, this is my undertsanding. Alex, do you agree on that ?
>
> * There are some tests which call session operations directly, these
> should be wrapped in txn begin/end.
ok.
>
> *I put some tests to AVL and jdbm partitions to test txns without the
> effect of interceptors and logical caches. These are multithreaded and
> are good to test the robustness of the data change and query layer and
> txn subsystem. Add more tests like this.
Ok.
>
> * Handle logical caches. As we talked handle these with read write
> locks taking cleanup in  case of an aborted txn into account.
>
> Please let me know if anyone wants to attack any of these. I might
> provide more input if needed.

I will spend some time by the end of this year on those things. We need 
to get a branch that builds and passes tests with flying colors, even if 
it's not perfect.

Also I see that there are TODO into the DefaultOperationExecutionManager 
class :

     public void delete( Partition partition, DeleteOperationContext 
deleteContext ) throws LdapException
     {
         Dn dn = deleteContext.getDn();

         //TODO TODO TODO REMOVE THIS
         partition.delete( deleteContext );

What should we do ?


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Txn Work Update(2)

[Selcuk AYA <ay...@gmail.com>]

Hi all,

update on my status:

* There is going to be a big change in my life soon and I will do a
big move. So I will be MIA for some time( around 2 months) and after
that I will have less time to contribute but I will try to keep being
involved in development and discussions.

* I did a recent check in. There are some tests that fail. I will list
them below. The basic reason that those tests are failing is that they
do queries and updates of which  some are transactional, some are not
and this leads to wrong results. And the basic reason they do queries
and updates in a non transactional way is that I did not put
transaction begin/commif/abort( ie, txn demarcation) at CoreSession.
This is the layer we start the execution of  the basic ldap operations
but I thought there might be clients of CoreSession which want to
group operations together and execute them as a single txn(If you
think this is not necessary, please let me know). Also, there is at
least search, for which the cursor is kept open above CoreSession, So
for search, the caller has to handle txn commit, abort.Also there
might be some cases where operations do not use session at all.


As a result, I decided to handle each  caller of CoreSession
operations separately and this increased the number changes that had
to be done. I handled ldap protocol handlers and change password.
Right now StockServerSuite is passing except three tests.
StockCoreSuite and MigratedStockCoreSuite is failing. The reason is
they use LDAPCoreConnection and this is not transactional yet.

For anybody who wants to get involved, here is the list of todo:
* Figure out why SaslBind and DelegatedAuthIT in stockserver suite is
failing. We get a KRBException here and I do not know this module well
enough.

*Handle LdapCoreConnection. This is used in applications which embed
apacheds i guess? The basic difficulty is that users of this class get
a cursor and the transaction should be comitted or aborted only when
this cursor closes . Moreover, when a cursor is open, user of
ldapcoreconnection can call other ldap operations and it might make
sense to group these in a transaction. If this is true, then it should
be the responsibility of the callers of LdapCoreSession to start and
end txns. If we are not going to let the user do this, we need to
store a pointer to txn context in the cursor and end the txn when
cursor is closed taking into account that the txn might have to be
upgraded to a readwrite txn from a read only txn if the user does
changess while holding the cursor open.

* There are some tests which call session operations directly, these
should be wrapped in txn begin/end.

*I put some tests to AVL and jdbm partitions to test txns without the
effect of interceptors and logical caches. These are multithreaded and
are good to test the robustness of the data change and query layer and
txn subsystem. Add more tests like this.

* Handle logical caches. As we talked handle these with read write
locks taking cleanup in  case of an aborted txn into account.

Please let me know if anyone wants to attack any of these. I might
provide more input if needed.

regards
Selcuk

On Thu, Dec 1, 2011 at 10:47 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 12/1/11 8:37 AM, Selcuk AYA wrote:
>>
>> On Wed, Nov 30, 2011 at 3:26 PM, Emmanuel Lecharny<el...@gmail.com>
>>  wrote:
>>>
>>> On 11/30/11 11:48 AM, Selcuk AYA wrote:
>>>>
>>>> Hi all,
>>>
>>>
>>> hi !
>>>
>>>> time to give an update on the txn status. I just checked in some code
>>>> with explanation of the changes for each project separately. Here is
>>>> just an overall summary and todo:
>>>>
>>>> *I added txn being/commit/abort to ldap requests and ran some test
>>>> with txns enabled. Put in various fixes to the txn layer based on the
>>>> tests. So first signs life signs for txns. More tests will be done but
>>>> I need to digress into AVL first. I also need to handle paged search.
>>>
>>> Great !
>>>
>>> Paged search is just a a normal search, but one that last longer. The way
>>> we
>>> handle it is by using a counter of sent responses.
>>>
>>> Basically, entries are sent one by one to the client, using a
>>> SearchResultEntry to hold the entry. With the pagedSearch, once we have
>>> reached the limit set by the client, we stop the sending of entries, and
>>> hold the cursor in the user's session, so that he can send back a new
>>> search
>>> request for then next entries - if needed -. Is there anything special
>>> you
>>> need to deal with here ? (btw, the cursor will be deleted as soon as the
>>> session will be closed. I hope this cleanup is effective...)
>>>
>> There is not much todo here as you guessed. The only problem is the
>> transaction context is stored in a thread local variable. When paged
>> search send partial data, we should store the context somewhere and
>> hand it off to the thread that will continue processing the search.
>
> ThradLocal is ok during the duration of the request. Why don't we use a
> session context that is associated with the user ? Right now, we are using a
> Context for each request, but inside it, we store the IoSession, where we
> can store some user's specific information, instead of storing them in a
> ThreadLocal.
>
>
>
>>
>>>> *I have also been working on removing interfaces related to
>>>> modification from the partition
>>>
>>> Hmmm, can you be a bit more specific ?
>>
>> When I say modification or change, I mean
>> modify/move/rename/moveAndRename/add/delete. Instead of partitions
>> implementing these separately, there is a layer above partitions which
>> handle these operations transactionally. The class that handles this
>> is called OperationExecutionManager. Partitions expose master table
>> and indices for the operation execution manager execute the
>> operations.
>
> Ok, understood. Fine with me. It will help a lot when it comes to write a
> new partition, instead of having to implement all the logic for each of
> these operations. Good move.
>
>>
>>>
>>>> and have been trying to get rid of the
>>>> Store interface. Got rid of the store interface except for config
>>>> methods and calls from the ui layer. I removed the modifications up
>>>> but still have to call the modification methods of the partition
>>>> interface because SchemaPartition triggers some changes based on the
>>>> logical  change. I believe those changes can be removed to schema
>>>> interceptor?
>>>
>>> We can't modify a schema. We can only add or delete a schema entry. This
>>> is
>>> a limitation, and I think it's mandated by the spec (don't remember where
>>> it
>>> comes from, but the idea is that modifying a schema element might imply a
>>> global modification on all the entries, something we don't want to
>>> accept).
>>>
>>>
>>>> In general, if a partition wants to do a change based on
>>>> a data update, then it should be able to do it during master table
>>>> update or by registering an interceptor(For example I changed multi
>>>> file ldif partition to do file updates when master table is updated).
>>>> Then we can get rid of the modification methods on partitions.
>>>
>>> you mean, the modify/move/rename/moveAndRename methods ?
>>
>> As mentioned above, by change or modification, I mean any operation
>> that ends up in update of data. For schema partitions, I believe
>> changes triggered after logical ldap change operations can be moved to
>> an interceptor.
>
> For schema, we now update the LDIF partition, whch means we go down to the
> partition layer, and everything else is managed by the interceptor, already
> (at least in trunk, and I think I have merged it back to your branch. Will
> check today).
>
>
>>
>>>> * Would be good to have statement snapshot isolation for
>>>> EntryFilteringCursor. Not having this doesnt affect ldap requests but
>>>> might affect code that works with EntryFilteringCursor directly.
>>>> Currently there are few cases of this especially during startup.
>>>
>>>
>>> I'll buy some more explanation : what is a 'statement snapshot' ?
>>
>> Say you do a search and get an entry filtering cursor and then you
>> modify the entries that you get from the cursor. Something like this:
>>
>> beginTransaction()
>> cursor = ..
>> while ( cursor.next() )
>> {
>>   entry = cursor.get();
>>   modify entry.
>> }
>>
>> endTransaction()
>>
>> If we have statement snapshot for the search, cursor will not see any
>> updates to the master table of index tables done through the
>> modifications.  This might be useful in some cases.
>
> Ok, got it.
>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>

Re: Txn Work Update(2)

[Emmanuel Lecharny <el...@gmail.com>]

On 12/1/11 8:37 AM, Selcuk AYA wrote:
> On Wed, Nov 30, 2011 at 3:26 PM, Emmanuel Lecharny<el...@gmail.com>  wrote:
>> On 11/30/11 11:48 AM, Selcuk AYA wrote:
>>> Hi all,
>>
>> hi !
>>
>>> time to give an update on the txn status. I just checked in some code
>>> with explanation of the changes for each project separately. Here is
>>> just an overall summary and todo:
>>>
>>> *I added txn being/commit/abort to ldap requests and ran some test
>>> with txns enabled. Put in various fixes to the txn layer based on the
>>> tests. So first signs life signs for txns. More tests will be done but
>>> I need to digress into AVL first. I also need to handle paged search.
>> Great !
>>
>> Paged search is just a a normal search, but one that last longer. The way we
>> handle it is by using a counter of sent responses.
>>
>> Basically, entries are sent one by one to the client, using a
>> SearchResultEntry to hold the entry. With the pagedSearch, once we have
>> reached the limit set by the client, we stop the sending of entries, and
>> hold the cursor in the user's session, so that he can send back a new search
>> request for then next entries - if needed -. Is there anything special you
>> need to deal with here ? (btw, the cursor will be deleted as soon as the
>> session will be closed. I hope this cleanup is effective...)
>>
> There is not much todo here as you guessed. The only problem is the
> transaction context is stored in a thread local variable. When paged
> search send partial data, we should store the context somewhere and
> hand it off to the thread that will continue processing the search.
ThradLocal is ok during the duration of the request. Why don't we use a 
session context that is associated with the user ? Right now, we are 
using a Context for each request, but inside it, we store the IoSession, 
where we can store some user's specific information, instead of storing 
them in a ThreadLocal.


>
>>> *I have also been working on removing interfaces related to
>>> modification from the partition
>> Hmmm, can you be a bit more specific ?
> When I say modification or change, I mean
> modify/move/rename/moveAndRename/add/delete. Instead of partitions
> implementing these separately, there is a layer above partitions which
> handle these operations transactionally. The class that handles this
> is called OperationExecutionManager. Partitions expose master table
> and indices for the operation execution manager execute the
> operations.
Ok, understood. Fine with me. It will help a lot when it comes to write 
a new partition, instead of having to implement all the logic for each 
of these operations. Good move.
>
>>
>>> and have been trying to get rid of the
>>> Store interface. Got rid of the store interface except for config
>>> methods and calls from the ui layer. I removed the modifications up
>>> but still have to call the modification methods of the partition
>>> interface because SchemaPartition triggers some changes based on the
>>> logical  change. I believe those changes can be removed to schema
>>> interceptor?
>> We can't modify a schema. We can only add or delete a schema entry. This is
>> a limitation, and I think it's mandated by the spec (don't remember where it
>> comes from, but the idea is that modifying a schema element might imply a
>> global modification on all the entries, something we don't want to accept).
>>
>>
>>> In general, if a partition wants to do a change based on
>>> a data update, then it should be able to do it during master table
>>> update or by registering an interceptor(For example I changed multi
>>> file ldif partition to do file updates when master table is updated).
>>> Then we can get rid of the modification methods on partitions.
>> you mean, the modify/move/rename/moveAndRename methods ?
> As mentioned above, by change or modification, I mean any operation
> that ends up in update of data. For schema partitions, I believe
> changes triggered after logical ldap change operations can be moved to
> an interceptor.
For schema, we now update the LDIF partition, whch means we go down to 
the partition layer, and everything else is managed by the interceptor, 
already (at least in trunk, and I think I have merged it back to your 
branch. Will check today).

>
>>> * Would be good to have statement snapshot isolation for
>>> EntryFilteringCursor. Not having this doesnt affect ldap requests but
>>> might affect code that works with EntryFilteringCursor directly.
>>> Currently there are few cases of this especially during startup.
>>
>> I'll buy some more explanation : what is a 'statement snapshot' ?
> Say you do a search and get an entry filtering cursor and then you
> modify the entries that you get from the cursor. Something like this:
>
> beginTransaction()
> cursor = ..
> while ( cursor.next() )
> {
>    entry = cursor.get();
>    modify entry.
> }
>
> endTransaction()
>
> If we have statement snapshot for the search, cursor will not see any
> updates to the master table of index tables done through the
> modifications.  This might be useful in some cases.
Ok, got it.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com