You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Alex Karasulu <ak...@apache.org> on 2011/11/01 04:35:49 UTC
Re: TXN work update
On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
Excuse the late response. Better late than never right ...
SNIP ...
> *Master table, index and index cursor wrappers: These provide
> transactionally consistent views of entries and index data. They merge
> what they read from their wrapped objects with the txn log manager. To
> make this easier, txn manager keeps track of index changes for a
> read/write txn in a summary form. One thing to note is that not all
> information that wrappers expose is transactionall consistent. One
> example is count methods. No need to %100 accurate on this. For these
> cases, the wrapper just calls the wrapped object.
>
>
Makes perfect sense to not bother with the count methods: slight variations
will not at all impact the optimizer's results.
> *For all the above, interfaces are defined in core-api and
> implementations and internal interfaces are in core. Also moved,
> index, indexcursor, table and related interfaces and helper classes to
> core-api from xdbm-partition.
>
>
I guess the xdbm-partition Maven module is now going to be pretty damn thin
or non-existent. Was this module destroyed and if not what actually remains?
> The todo is the following:
> *add changes to keep track of dn space changes.
> *test txn manager services
> *move the modification code path in AbstractBTreePartition in xdbm to
> high up to core. Probably modifications should be done in
> partitionNexus and partition nexus should handle preparing txn log
> edits and wal them. The overall flow for modifications will be:
> -DefaultDirectoryService:
> -begintxn
> -execute interceptor chain
> - handle txn abort, or conflict.
>
>
Moving this into the PartitionNexus might not be a good idea but no problem
for now we can move it later. Let me explain why:
Eventually we're going to enable a root Partition with Partition nesting
and so when this happens the PartitionNexus will just be another nestable
Partition since these will have to handle routing based on DN to other
partitions residing/nested under it.
I see two possible locations for this functionality:
(1) Let the InterceptorChain itself handle this since it can demarcate the
start and end of calls into the chain with Txn begin and abort/commit
calls. It does this by making calls against the TxnManager which I guess is
the entire facade for the transaction subsystem.
---OR---
(2) Handle Txn demarcation within the CoreSession. However this might not
be optimal due to the need to handle additional logic which might be
required for handling chain re-entry concerns.
NOTE: I've not actually looked at the code after these major moves so my
advice might not be very dependable. I will try to setup my environment to
get a better idea of these matters.
However for the time being do whatever actually makes this thing work.
Let's follow an agile methodology. This thing is big. So let's get it
working with solid test coverage then we can actually look at shuffling
things around to optimal positions. Not saying what you've chosen is wrong
... it might just present the need for some additional refactoring when
other features might need to be introduced.
These are some of the biggest changes to the architecture to have taken
place in years and you're doing a great job.
*move xdbm-search to core as well. Making search transactional will
> mostly be mostly mechanica after this point I think(hope). It should
> just use the wrappers the txn manager provides for index, master and
> cursors it gets from the partitions.
>
> *handle caches various interceptors keep. I am thinking of handling
> this with a common read-write lock.
>
>
This was the latest issue for which I see some more threads. Will look at
that as well.
Again sorry about late responses.
Regards,
Alex
> regards
> Selcuk
>
> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA <ay...@gmail.com> wrote:
> > Hi all,
> >
> > Emmanuel wants to update the txns branch and I think it is a good time
> > to give a good status update. Following is what I have been doing and
> > the plan:
> >
> > *Log Manager: A general log manager that can be used by txns as well.
> > Note that these logs are on disk. This is tested.
> >
> > *Txn Manager:Most of the services that should be provided by the txn
> > manager is coded:
> > begin, commit, abort, keeping
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
On Tue, Nov 1, 2011 at 1:19 PM, Selcuk AYA <ay...@gmail.com> wrote:
> thanks for the feedback! Please see inline:
>
> On Tue, Nov 1, 2011 at 5:35 AM, Alex Karasulu <ak...@apache.org>
> wrote:
> > On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
> >
> > Excuse the late response. Better late than never right ...
> > SNIP ...
> >
> >>
> >> *Master table, index and index cursor wrappers: These provide
> >> transactionally consistent views of entries and index data. They merge
> >> what they read from their wrapped objects with the txn log manager. To
> >> make this easier, txn manager keeps track of index changes for a
> >> read/write txn in a summary form. One thing to note is that not all
> >> information that wrappers expose is transactionall consistent. One
> >> example is count methods. No need to %100 accurate on this. For these
> >> cases, the wrapper just calls the wrapped object.
> >>
> >
> > Makes perfect sense to not bother with the count methods: slight
> variations
> > will not at all impact the optimizer's results.
> >
> >>
> >> *For all the above, interfaces are defined in core-api and
> >> implementations and internal interfaces are in core. Also moved,
> >> index, indexcursor, table and related interfaces and helper classes to
> >> core-api from xdbm-partition.
> >>
> >
> > I guess the xdbm-partition Maven module is now going to be pretty damn
> thin
> > or non-existent. Was this module destroyed and if not what actually
> remains?
>
> I think some util classes for tree based implementations remain there.
> Also there some avl related classes.
>
> >
> >>
> >> The todo is the following:
> >> *add changes to keep track of dn space changes.
> >> *test txn manager services
> >> *move the modification code path in AbstractBTreePartition in xdbm to
> >> high up to core. Probably modifications should be done in
> >> partitionNexus and partition nexus should handle preparing txn log
> >> edits and wal them. The overall flow for modifications will be:
> >> -DefaultDirectoryService:
> >> -begintxn
> >> -execute interceptor chain
> >> - handle txn abort, or conflict.
> >>
> >
> > Moving this into the PartitionNexus might not be a good idea but no
> problem
> > for now we can move it later. Let me explain why:
> > Eventually we're going to enable a root Partition with Partition nesting
> and
> > so when this happens the PartitionNexus will just be another nestable
> > Partition since these will have to handle routing based on DN to other
> > partitions residing/nested under it.
> > I see two possible locations for this functionality:
> > (1) Let the InterceptorChain itself handle this since it can demarcate
> the
> > start and end of calls into the chain with Txn begin and abort/commit
> calls.
> > It does this by making calls against the TxnManager which I guess is the
> > entire facade for the transaction subsystem.
> > ---OR---
> > (2) Handle Txn demarcation within the CoreSession. However this might
> not be
> > optimal due to the need to handle additional logic which might be
> required
> > for handling chain re-entry concerns.
> > NOTE: I've not actually looked at the code after these major moves so my
> > advice might not be very dependable. I will try to setup my environment
> to
> > get a better idea of these matters.
> > However for the time being do whatever actually makes this thing work.
> Let's
> > follow an agile methodology. This thing is big. So let's get it working
> with
> > solid test coverage then we can actually look at shuffling things around
> to
> > optimal positions. Not saying what you've chosen is wrong ... it might
> just
> > present the need for some additional refactoring when other features
> might
> > need to be introduced.
> > These are some of the biggest changes to the architecture to have taken
> > place in years and you're doing a great job.
>
> You are right for txn demarcation. I wrote it wrong in the email.
> Demarcation has to be done either at defaultcoresession or
> defaultoperationmanager.
>
> as for where to handle the change logic, as you mentioned we need a
> place where all interceptor chain routes end up for modification and
> ideally we should handle modification logic above partitions using
> master table and index interfaces so that we have a common place to
> prepare and apply txn log edits. PartitionNexus seems to fit this
> requirement for now. If we add another layer above PartitionNexus
> which can get master and index table from below layers and work with
> them, we should be able to move the change logic up there.
>
> >>
> >> *move xdbm-search to core as well. Making search transactional will
> >> mostly be mostly mechanica after this point I think(hope). It should
> >> just use the wrappers the txn manager provides for index, master and
> >> cursors it gets from the partitions.
> >>
> >> *handle caches various interceptors keep. I am thinking of handling
> >> this with a common read-write lock.
> >>
> >
> > This was the latest issue for which I see some more threads. Will look at
> > that as well.
> This is the issue I talked you about briefly. These are mostly admin
> caches that change infrequently and that are read mostly. The
> difficulty with them is that they are not always entry caches. They
> might map Dns to some logical property of the entry. A simple and good
> example is notaliascache (whether we really need this is open to
> debate but we have it now). This maps a Dn to whether the entry is an
> alias. If we followed the normal way of merging what is read from
> partitions with the txn log, we would have to a way of merging what is
> read from this notaliascache with the log. This is not very difficult
> but as we have quite a number of these caches, having a separate merge
> and update logic for each of them is a pain and error prone.
>
> Instead, what I thought was to use a single system wide read-write
> lock for these caches. Since they are read mostly, a txn might update
> its lock to exclusive when it needs to change one of these caches and
> might release it when it actually commits/aborts.
>
>
Yep I remember this conversation very well. I think this is the best route
for us to take for the time being. Handling each cache separate takes more
effort and it might not pay off so let's not do unnecessary
write-ahead-coding :-D.
Who knows it might be the best way for us to proceed from all aspects
including performance. You just can't tell these things until you test the
server in a stress test environment. That's why we need a solid test
environment.
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/2/11 10:10 AM, Selcuk AYA wrote:
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level? In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
>> Once at the nexus level, the elements won't be modified.
>> The ChangeLog stores (currently) data in memory, and those data are kept for
>> a while, so cross transactions. This is the reason why we clone the entry
>> and Attributes.
> with transactions, the attributes and modifications will be kept in
> memory cross transactions as well. But my understanding( and the thing
> I wanted to verify) is that Attributes, their Values and Modification
> objects should not change any more once they are at the Nexus level as
> these objects created per modification request and they are ready to
> be applied to Entry objects.
Once in the nexus, the objects aren't modified.
About the changelog, you have to know that we keep them as long as we
need the elements, but as we keep all of them, we are memory bound
anyway. Also we don't need to keep the original entry in the changeLog,
as we only have to apply the revert operation. The original entry is
just needed to compute the reverse entry.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
On Wed, Nov 2, 2011 at 10:52 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/2/11 9:12 AM, Selcuk AYA wrote:
>>
>> Hi,
>> a few points and questions:
>> *I am planning to use a common ID for all partitions. I checked Hbase
>> partition and it uses UUID as the ID. Is it ok to use this for all
>> partitions?
>
> Yes. This should have been done a long time ago.
>>
>> *It seems that I need to move txn and log package implementations to
>> core-shared because of the recent reorg. Coresession and nexus
>> implementation is moved there and interceptors might need these
>> packages.
>
> Yep. We haven't yet merged core-shared with core, but atm, core-shared is
> the place to put new classes.
>>
>> * When preparing txn log edits, I will need to log Modification and
>> Attribute objects and keep them in memory for a while. Is it necessary
>> to clone these objects while doing the logging at partition nexus
>> level? In other words, will they be modified after reaching nexus
>> level. I am thinking no but I wanted to check because I looked at
>> changelog implementation and it does cloning.
>
> Once at the nexus level, the elements won't be modified.
> The ChangeLog stores (currently) data in memory, and those data are kept for
> a while, so cross transactions. This is the reason why we clone the entry
> and Attributes.
with transactions, the attributes and modifications will be kept in
memory cross transactions as well. But my understanding( and the thing
I wanted to verify) is that Attributes, their Values and Modification
objects should not change any more once they are at the Nexus level as
these objects created per modification request and they are ready to
be applied to Entry objects.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/2/11 9:12 AM, Selcuk AYA wrote:
> Hi,
> a few points and questions:
> *I am planning to use a common ID for all partitions. I checked Hbase
> partition and it uses UUID as the ID. Is it ok to use this for all
> partitions?
Yes. This should have been done a long time ago.
>
> *It seems that I need to move txn and log package implementations to
> core-shared because of the recent reorg. Coresession and nexus
> implementation is moved there and interceptors might need these
> packages.
Yep. We haven't yet merged core-shared with core, but atm, core-shared
is the place to put new classes.
>
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level? In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
Once at the nexus level, the elements won't be modified.
The ChangeLog stores (currently) data in memory, and those data are kept
for a while, so cross transactions. This is the reason why we clone the
entry and Attributes.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/14/11 9:09 AM, Selcuk AYA wrote:
> hi,
Hi !
> i am currently working on changing modification layer to use txn
> services and have a few questions about dropaliasindices:
>
> *is it true that an entry has object class ALIAS_OC iff it contains
> attribute ALIASED_OBJECT_NAME_AT.
The Alias ObjectClass has one mandatory AttributeType, aliasedObjectName :
objectclass ( 2.5.6.1
NAME 'alias'
DESC 'RFC2256: an alias'
SUP top
STRUCTURAL
MUST aliasedObjectName
)
>
> *where can I check the ldap definitions of atttributes given in
> SchemaConstants. I wanted check if ALIASED_OBJECT_NAME_AT is single
> valued.
The aliasedObjectName is single valued :
attributetype ( 2.5.4.1
NAME ( 'aliasedObjectName' 'aliasedEntryName' )
DESC 'RFC2256: name of aliased object'
EQUALITY distinguishedNameMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.12
SINGLE-VALUE
)
The easiest way to get those information is not to google them, but to
use the schema editor in Apache Directory Studio :)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
hi,
i am currently working on changing modification layer to use txn
services and have a few questions about dropaliasindices:
*is it true that an entry has object class ALIAS_OC iff it contains
attribute ALIASED_OBJECT_NAME_AT.
*where can I check the ldap definitions of atttributes given in
SchemaConstants. I wanted check if ALIASED_OBJECT_NAME_AT is single
valued.
thanks
Selcuk
On Thu, Nov 10, 2011 at 10:51 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/10/11 7:56 AM, Selcuk AYA wrote:
>>
>> Hi,
>> I have made the changes to use UUID for ID. I have also removed some
>> unnecessary type parameters. In particular, master table is not
>> generic anymore. For index and index cursors, only value type is a
>> generic parameter. Search engine and partitions and tests are changes
>> accordingly. Note that counter maintained for ID generation in Btree
>> partitions are not necessary anymore. Also, admin table for Jdbm
>> parition is removed. One thing I noticed in the tests were implicit
>> assumption of master table and indices with the key ID having the same
>> order as the insertion order of entries. This assumption is not valid
>> anymore as UUID generation is random.
>
> Great job !
>>
>> There is one outstanding issue. Attribute types for system indices
>> having UUID as the key has integer comparator as their ordering rule
>> in the LDAP attribute type definition. This should be changed to use
>> UUID ldap comparator. This is not done yet as I dont know how to do
>> it. Instead, we check for specific attribute types during index
>> initializations and use UUID comparator if necessary.
>
> I can handle that.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/10/11 7:56 AM, Selcuk AYA wrote:
> Hi,
> I have made the changes to use UUID for ID. I have also removed some
> unnecessary type parameters. In particular, master table is not
> generic anymore. For index and index cursors, only value type is a
> generic parameter. Search engine and partitions and tests are changes
> accordingly. Note that counter maintained for ID generation in Btree
> partitions are not necessary anymore. Also, admin table for Jdbm
> parition is removed. One thing I noticed in the tests were implicit
> assumption of master table and indices with the key ID having the same
> order as the insertion order of entries. This assumption is not valid
> anymore as UUID generation is random.
Great job !
>
> There is one outstanding issue. Attribute types for system indices
> having UUID as the key has integer comparator as their ordering rule
> in the LDAP attribute type definition. This should be changed to use
> UUID ldap comparator. This is not done yet as I dont know how to do
> it. Instead, we check for specific attribute types during index
> initializations and use UUID comparator if necessary.
I can handle that.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
Hi,
I have made the changes to use UUID for ID. I have also removed some
unnecessary type parameters. In particular, master table is not
generic anymore. For index and index cursors, only value type is a
generic parameter. Search engine and partitions and tests are changes
accordingly. Note that counter maintained for ID generation in Btree
partitions are not necessary anymore. Also, admin table for Jdbm
parition is removed. One thing I noticed in the tests were implicit
assumption of master table and indices with the key ID having the same
order as the insertion order of entries. This assumption is not valid
anymore as UUID generation is random.
There is one outstanding issue. Attribute types for system indices
having UUID as the key has integer comparator as their ordering rule
in the LDAP attribute type definition. This should be changed to use
UUID ldap comparator. This is not done yet as I dont know how to do
it. Instead, we check for specific attribute types during index
initializations and use UUID comparator if necessary.
Currently all tests are passing.
regards,
Selcuk
On Fri, Nov 4, 2011 at 9:02 PM, Stefan Seelmann <se...@apache.org> wrote:
> On Fri, Nov 4, 2011 at 9:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
>> I discussed using generics for ID with Emmanuel a little more. The
>> related change is mostly mechanical but it is a big change so I would
>> like to see if people have any opinion. Remember that alongside with
>> transaction changes, we are going towards a generic transactional
>> modification layer on top of partitions and a transactional search
>> engine that can work with any partition. Default search engine
>> achieves what we want in this sense but it works with Store interface
>> rather than Partition interface. Default Search Engine is designed to
>> work with generic ID. Store interface also uses generic ID. Now, if we
>> want to move DefaultSearchEngine to core and want it to work with
>> Partition interfaces, we have to options:
>> 1)Make partition interface parametrized too. Change it to
>> Partition<ID> and make it expose master tables and indices using
>> generic ID. DefaultSearch engine can currently work with such an
>> interface. We will also make the transactional modification layer use
>> this parametrized interface.We will also need to implement partitions
>> using the generic ID type rather than Long or UUID
>>
>> 2) Use UUID all over rather than generics. Basically get rid of
>> generic ID type. Change search engine to use UUID as well.
>
> +1 to the 2nd option.
>
> I introduced the generid <ID> two years ago [1] while I as was
> implementing the HBase partition prototype. Prior to that the ID was a
> Long which didn't work well with HBase. Now that we all agree that
> UUID should be used I see no reason why the generic ID should be kept.
>
> Kind Regards,
> Stefan
>
> [1] https://issues.apache.org/jira/browse/DIRSERVER-1472
>
Re: TXN work update
Posted by Stefan Seelmann <se...@apache.org>.
On Fri, Nov 4, 2011 at 9:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
+1 to the 2nd option.
I introduced the generid <ID> two years ago [1] while I as was
implementing the HBase partition prototype. Prior to that the ID was a
Long which didn't work well with HBase. Now that we all agree that
UUID should be used I see no reason why the generic ID should be kept.
Kind Regards,
Stefan
[1] https://issues.apache.org/jira/browse/DIRSERVER-1472
Re: TXN work update
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 4 nov. 2011, at 10:24, Alex Karasulu wrote:
> I vote for the second option and to get rid of Generics on these interfaces. Why?
>
> (1) We need to go to UUIDs since we will soon be living in a replicated world and there's no point to a partition specific id.
> (2) We can still do (1) with a Long but why bother?
> (3) Using a UUID as the PK for entries gets rid of the need for the UUID index since the master automatically becomes the UUID index.
> (4) Generics have made everything way too complex in this region of the code and I do not foresee a need to use anything else at this point in time. If the need arises we can consider it after everything is working ... a simple maneuver/re-factoring.
+1
Although I like Generics a lot, it makes probably more sense to avoid them here in this particular context.
Regards,
Pierre-Arnaud
>
> Regards,
> Alex
>
> On Fri, Nov 4, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
>
>
> First option has the flexibility of changing ID to a different type
> but I am not sure it makes sense to change the ID from release to
> release. I am inclined to go with the second option at the moment.
>
> regards,
> Selcuk
>
> On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> > On 11/3/11 4:44 PM, Selcuk AYA wrote:
> >>
> >> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
> >> wrote:
> >>>
> >>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
> >>>>
> >>>> Hi,
> >>>> a few points and questions:
> >>>> *I am planning to use a common ID for all partitions. I checked Hbase
> >>>> partition and it uses UUID as the ID. Is it ok to use this for all
> >>>> partitions?
> >>>>
> >>> This great and something we wanted to do for a very long time as Emmanuel
> >>> stated.
> >>
> >> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> >> am making partitions expose their master tabe and indices. For making
> >> all partitions use UUID:
> >> * I can change partitions to work with UUID directly and make them
> >> return master tables as MasterTable<Entry,UUID> and index tables in a
> >> similar way
> >> *or I can paratmetrize partition interface like Partition<ID> . They
> >> can return master table as<MasterTable,ID> and index tables in a
> >> similar way. This is looks to me the right way but it eventually leads
> >> to more changes.
> >>
> >> which one do you think is better?
> >
> > Let's not use generic when it's not needed...
> >
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
>
>
> --
> Best Regards,
> -- Alex
>
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
I vote for the second option and to get rid of Generics on these
interfaces. Why?
(1) We need to go to UUIDs since we will soon be living in a replicated
world and there's no point to a partition specific id.
(2) We can still do (1) with a Long but why bother?
(3) Using a UUID as the PK for entries gets rid of the need for the UUID
index since the master automatically becomes the UUID index.
(4) Generics have made everything way too complex in this region of the
code and I do not foresee a need to use anything else at this point in
time. If the need arises we can consider it after everything is working ...
a simple maneuver/re-factoring.
Regards,
Alex
On Fri, Nov 4, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
>
>
> First option has the flexibility of changing ID to a different type
> but I am not sure it makes sense to change the ID from release to
> release. I am inclined to go with the second option at the moment.
>
> regards,
> Selcuk
>
> On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> > On 11/3/11 4:44 PM, Selcuk AYA wrote:
> >>
> >> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
> >> wrote:
> >>>
> >>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>> a few points and questions:
> >>>> *I am planning to use a common ID for all partitions. I checked Hbase
> >>>> partition and it uses UUID as the ID. Is it ok to use this for all
> >>>> partitions?
> >>>>
> >>> This great and something we wanted to do for a very long time as
> Emmanuel
> >>> stated.
> >>
> >> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> >> am making partitions expose their master tabe and indices. For making
> >> all partitions use UUID:
> >> * I can change partitions to work with UUID directly and make them
> >> return master tables as MasterTable<Entry,UUID> and index tables in a
> >> similar way
> >> *or I can paratmetrize partition interface like Partition<ID> . They
> >> can return master table as<MasterTable,ID> and index tables in a
> >> similar way. This is looks to me the right way but it eventually leads
> >> to more changes.
> >>
> >> which one do you think is better?
> >
> > Let's not use generic when it's not needed...
> >
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
I discussed using generics for ID with Emmanuel a little more. The
related change is mostly mechanical but it is a big change so I would
like to see if people have any opinion. Remember that alongside with
transaction changes, we are going towards a generic transactional
modification layer on top of partitions and a transactional search
engine that can work with any partition. Default search engine
achieves what we want in this sense but it works with Store interface
rather than Partition interface. Default Search Engine is designed to
work with generic ID. Store interface also uses generic ID. Now, if we
want to move DefaultSearchEngine to core and want it to work with
Partition interfaces, we have to options:
1)Make partition interface parametrized too. Change it to
Partition<ID> and make it expose master tables and indices using
generic ID. DefaultSearch engine can currently work with such an
interface. We will also make the transactional modification layer use
this parametrized interface.We will also need to implement partitions
using the generic ID type rather than Long or UUID
2) Use UUID all over rather than generics. Basically get rid of
generic ID type. Change search engine to use UUID as well.
First option has the flexibility of changing ID to a different type
but I am not sure it makes sense to change the ID from release to
release. I am inclined to go with the second option at the moment.
regards,
Selcuk
On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/3/11 4:44 PM, Selcuk AYA wrote:
>>
>> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
>> wrote:
>>>
>>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> a few points and questions:
>>>> *I am planning to use a common ID for all partitions. I checked Hbase
>>>> partition and it uses UUID as the ID. Is it ok to use this for all
>>>> partitions?
>>>>
>>> This great and something we wanted to do for a very long time as Emmanuel
>>> stated.
>>
>> I am working on JDBM and AVL partitions to make them usee UUID. Also I
>> am making partitions expose their master tabe and indices. For making
>> all partitions use UUID:
>> * I can change partitions to work with UUID directly and make them
>> return master tables as MasterTable<Entry,UUID> and index tables in a
>> similar way
>> *or I can paratmetrize partition interface like Partition<ID> . They
>> can return master table as<MasterTable,ID> and index tables in a
>> similar way. This is looks to me the right way but it eventually leads
>> to more changes.
>>
>> which one do you think is better?
>
> Let's not use generic when it's not needed...
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/3/11 4:44 PM, Selcuk AYA wrote:
> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org> wrote:
>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
>>> Hi,
>>> a few points and questions:
>>> *I am planning to use a common ID for all partitions. I checked Hbase
>>> partition and it uses UUID as the ID. Is it ok to use this for all
>>> partitions?
>>>
>> This great and something we wanted to do for a very long time as Emmanuel
>> stated.
> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> am making partitions expose their master tabe and indices. For making
> all partitions use UUID:
> * I can change partitions to work with UUID directly and make them
> return master tables as MasterTable<Entry,UUID> and index tables in a
> similar way
> *or I can paratmetrize partition interface like Partition<ID> . They
> can return master table as<MasterTable,ID> and index tables in a
> similar way. This is looks to me the right way but it eventually leads
> to more changes.
>
> which one do you think is better?
Let's not use generic when it's not needed...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu <ak...@apache.org> wrote:
> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
>>
>> Hi,
>> a few points and questions:
>> *I am planning to use a common ID for all partitions. I checked Hbase
>> partition and it uses UUID as the ID. Is it ok to use this for all
>> partitions?
>>
>
> This great and something we wanted to do for a very long time as Emmanuel
> stated.
I am working on JDBM and AVL partitions to make them usee UUID. Also I
am making partitions expose their master tabe and indices. For making
all partitions use UUID:
* I can change partitions to work with UUID directly and make them
return master tables as MasterTable<Entry,UUID> and index tables in a
similar way
*or I can paratmetrize partition interface like Partition<ID> . They
can return master table as <MasterTable,ID> and index tables in a
similar way. This is looks to me the right way but it eventually leads
to more changes.
which one do you think is better?
>
>>
>> *It seems that I need to move txn and log package implementations to
>> core-shared because of the recent reorg. Coresession and nexus
>> implementation is moved there and interceptors might need these
>> packages.
>>
>
> +1
>
>>
>> * When preparing txn log edits, I will need to log Modification and
>> Attribute objects and keep them in memory for a while. Is it necessary
>> to clone these objects while doing the logging at partition nexus
>> level?
>
> Don't think so.
>
>>
>> In other words, will they be modified after reaching nexus
>> level. I am thinking no but I wanted to check because I looked at
>> changelog implementation and it does cloning.
>>
>
> Thanks,
> Alex
>
>>
>> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com>
>> wrote:
>> > Hi !
>> >
>> > comments inline...
>> >
>> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
>> >>
>> >> thanks for the feedback! Please see inline:
>> >
>> > <snip/>
>> >>
>> >>
>> >> I guess the xdbm-partition Maven module is now going to be pretty damn
>> >> thin
>> >> or non-existent. Was this module destroyed and if not what actually
>> >> remains?
>> >> I think some util classes for tree based implementations remain there.
>> >> Also there some avl related classes.
>> >
>> > We can review this project and eventually move the remaining classes
>> > elsewhere.
>> >
>> >>
>> >>>> The todo is the following:
>> >>>> *add changes to keep track of dn space changes.
>> >>>> *test txn manager services
>> >>>> *move the modification code path in AbstractBTreePartition in xdbm to
>> >>>> high up to core. Probably modifications should be done in
>> >>>> partitionNexus and partition nexus should handle preparing txn log
>> >>>> edits and wal them. The overall flow for modifications will be:
>> >>>> -DefaultDirectoryService:
>> >>>> -begintxn
>> >>>> -execute interceptor chain
>> >>>> - handle txn abort, or conflict.
>> >>>>
>> >>> Moving this into the PartitionNexus might not be a good idea but no
>> >>> problem
>> >>> for now we can move it later. Let me explain why:
>> >>> Eventually we're going to enable a root Partition with Partition
>> >>> nesting
>> >>> and
>> >>> so when this happens the PartitionNexus will just be another nestable
>> >>> Partition since these will have to handle routing based on DN to other
>> >>> partitions residing/nested under it.
>> >>> I see two possible locations for this functionality:
>> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
>> >>> the
>> >>> start and end of calls into the chain with Txn begin and abort/commit
>> >>> calls.
>> >>> It does this by making calls against the TxnManager which I guess is
>> >>> the
>> >>> entire facade for the transaction subsystem.
>> >>> ---OR---
>> >>> (2) Handle Txn demarcation within the CoreSession. However this might
>> >>> not
>> >>> be
>> >>> optimal due to the need to handle additional logic which might be
>> >>> required
>> >>> for handling chain re-entry concerns.
>> >>> NOTE: I've not actually looked at the code after these major moves so
>> >>> my
>> >>> advice might not be very dependable. I will try to setup my
>> >>> environment
>> >>> to
>> >>> get a better idea of these matters.
>> >>> However for the time being do whatever actually makes this thing work.
>> >>> Let's
>> >>> follow an agile methodology. This thing is big. So let's get it
>> >>> working
>> >>> with
>> >>> solid test coverage then we can actually look at shuffling things
>> >>> around
>> >>> to
>> >>> optimal positions. Not saying what you've chosen is wrong ... it might
>> >>> just
>> >>> present the need for some additional refactoring when other features
>> >>> might
>> >>> need to be introduced.
>> >>> These are some of the biggest changes to the architecture to have
>> >>> taken
>> >>> place in years and you're doing a great job.
>> >>
>> >> You are right for txn demarcation. I wrote it wrong in the email.
>> >> Demarcation has to be done either at defaultcoresession or
>> >> defaultoperationmanager.
>> >
>> > OperationManager is probably the right place.
>> >
>> > For inner operations, I'm almost 100% sure they aren't modifiying
>> > anything
>> > (the only modification done as a inner operation has been removed a
>> > while
>> > ago : it was dealing with the addition of the ModifyTimeStamp and
>> > ModifiersName attribute, and it' snow done in the main operation). So we
>> > are
>> > safe if they are done in the current external transaction.
>> >
>> >>
>> >> as for where to handle the change logic, as you mentioned we need a
>> >> place where all interceptor chain routes end up for modification and
>> >> ideally we should handle modification logic above partitions using
>> >> master table and index interfaces so that we have a common place to
>> >> prepare and apply txn log edits. PartitionNexus seems to fit this
>> >> requirement for now. If we add another layer above PartitionNexus
>> >> which can get master and index table from below layers and work with
>> >> them, we should be able to move the change logic up there.
>> >>
>> >>>> *move xdbm-search to core as well. Making search transactional will
>> >>>> mostly be mostly mechanica after this point I think(hope). It should
>> >>>> just use the wrappers the txn manager provides for index, master and
>> >>>> cursors it gets from the partitions.
>> >>>>
>> >>>> *handle caches various interceptors keep. I am thinking of handling
>> >>>> this with a common read-write lock.
>> >>>>
>> >>> This was the latest issue for which I see some more threads. Will look
>> >>> at
>> >>> that as well.
>> >>
>> >> This is the issue I talked you about briefly. These are mostly admin
>> >> caches that change infrequently and that are read mostly. The
>> >> difficulty with them is that they are not always entry caches. They
>> >> might map Dns to some logical property of the entry. A simple and good
>> >> example is notaliascache (whether we really need this is open to
>> >> debate but we have it now).
>> >
>> > This is a very interesting point. The notAliasCache is definitively
>> > questionable. The question though is to find a better way to deal with
>> > it.
>> > An option would be to not support alias (yes, I know, pretty drastic,
>> > but if
>> > it allows us to speed up the delivery of a production ready server, I
>> > think
>> > this would be a good tradeoff. Also note that once we have a solid base,
>> > we
>> > can reintroduce alias handling in a future version).
>> >
>> >> This maps a Dn to whether the entry is an
>> >> alias. If we followed the normal way of merging what is read from
>> >> partitions with the txn log, we would have to a way of merging what is
>> >> read from this notaliascache with the log. This is not very difficult
>> >> but as we have quite a number of these caches, having a separate merge
>> >> and update logic for each of them is a pain and error prone.
>> >>
>> >> Instead, what I thought was to use a single system wide read-write
>> >> lock for these caches. Since they are read mostly, a txn might update
>> >> its lock to exclusive when it needs to change one of these caches and
>> >> might release it when it actually commits/aborts.
>> >
>> > We may start a new thread to discuss further those cache issues. We have
>> > many caches, for may different thinsg, and many different approaches to
>> > deal
>> > with the (MRU, EhCache, etc...). It's probably a good timing for
>> > clarifying
>> > this item...
>> >
>> > --
>> > Regards,
>> > Cordialement,
>> > Emmanuel Lécharny
>> > www.iktek.com
>> >
>> >
>
>
>
> --
> Best Regards,
> -- Alex
>
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> Hi,
> a few points and questions:
> *I am planning to use a common ID for all partitions. I checked Hbase
> partition and it uses UUID as the ID. Is it ok to use this for all
> partitions?
>
>
This great and something we wanted to do for a very long time as Emmanuel
stated.
> *It seems that I need to move txn and log package implementations to
> core-shared because of the recent reorg. Coresession and nexus
> implementation is moved there and interceptors might need these
> packages.
>
>
+1
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level?
Don't think so.
> In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
>
>
Thanks,
Alex
>
> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> > Hi !
> >
> > comments inline...
> >
> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
> >>
> >> thanks for the feedback! Please see inline:
> >
> > <snip/>
> >>
> >>
> >> I guess the xdbm-partition Maven module is now going to be pretty damn
> >> thin
> >> or non-existent. Was this module destroyed and if not what actually
> >> remains?
> >> I think some util classes for tree based implementations remain there.
> >> Also there some avl related classes.
> >
> > We can review this project and eventually move the remaining classes
> > elsewhere.
> >
> >>
> >>>> The todo is the following:
> >>>> *add changes to keep track of dn space changes.
> >>>> *test txn manager services
> >>>> *move the modification code path in AbstractBTreePartition in xdbm to
> >>>> high up to core. Probably modifications should be done in
> >>>> partitionNexus and partition nexus should handle preparing txn log
> >>>> edits and wal them. The overall flow for modifications will be:
> >>>> -DefaultDirectoryService:
> >>>> -begintxn
> >>>> -execute interceptor chain
> >>>> - handle txn abort, or conflict.
> >>>>
> >>> Moving this into the PartitionNexus might not be a good idea but no
> >>> problem
> >>> for now we can move it later. Let me explain why:
> >>> Eventually we're going to enable a root Partition with Partition
> nesting
> >>> and
> >>> so when this happens the PartitionNexus will just be another nestable
> >>> Partition since these will have to handle routing based on DN to other
> >>> partitions residing/nested under it.
> >>> I see two possible locations for this functionality:
> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
> >>> the
> >>> start and end of calls into the chain with Txn begin and abort/commit
> >>> calls.
> >>> It does this by making calls against the TxnManager which I guess is
> the
> >>> entire facade for the transaction subsystem.
> >>> ---OR---
> >>> (2) Handle Txn demarcation within the CoreSession. However this might
> not
> >>> be
> >>> optimal due to the need to handle additional logic which might be
> >>> required
> >>> for handling chain re-entry concerns.
> >>> NOTE: I've not actually looked at the code after these major moves so
> my
> >>> advice might not be very dependable. I will try to setup my environment
> >>> to
> >>> get a better idea of these matters.
> >>> However for the time being do whatever actually makes this thing work.
> >>> Let's
> >>> follow an agile methodology. This thing is big. So let's get it working
> >>> with
> >>> solid test coverage then we can actually look at shuffling things
> around
> >>> to
> >>> optimal positions. Not saying what you've chosen is wrong ... it might
> >>> just
> >>> present the need for some additional refactoring when other features
> >>> might
> >>> need to be introduced.
> >>> These are some of the biggest changes to the architecture to have taken
> >>> place in years and you're doing a great job.
> >>
> >> You are right for txn demarcation. I wrote it wrong in the email.
> >> Demarcation has to be done either at defaultcoresession or
> >> defaultoperationmanager.
> >
> > OperationManager is probably the right place.
> >
> > For inner operations, I'm almost 100% sure they aren't modifiying
> anything
> > (the only modification done as a inner operation has been removed a while
> > ago : it was dealing with the addition of the ModifyTimeStamp and
> > ModifiersName attribute, and it' snow done in the main operation). So we
> are
> > safe if they are done in the current external transaction.
> >
> >>
> >> as for where to handle the change logic, as you mentioned we need a
> >> place where all interceptor chain routes end up for modification and
> >> ideally we should handle modification logic above partitions using
> >> master table and index interfaces so that we have a common place to
> >> prepare and apply txn log edits. PartitionNexus seems to fit this
> >> requirement for now. If we add another layer above PartitionNexus
> >> which can get master and index table from below layers and work with
> >> them, we should be able to move the change logic up there.
> >>
> >>>> *move xdbm-search to core as well. Making search transactional will
> >>>> mostly be mostly mechanica after this point I think(hope). It should
> >>>> just use the wrappers the txn manager provides for index, master and
> >>>> cursors it gets from the partitions.
> >>>>
> >>>> *handle caches various interceptors keep. I am thinking of handling
> >>>> this with a common read-write lock.
> >>>>
> >>> This was the latest issue for which I see some more threads. Will look
> at
> >>> that as well.
> >>
> >> This is the issue I talked you about briefly. These are mostly admin
> >> caches that change infrequently and that are read mostly. The
> >> difficulty with them is that they are not always entry caches. They
> >> might map Dns to some logical property of the entry. A simple and good
> >> example is notaliascache (whether we really need this is open to
> >> debate but we have it now).
> >
> > This is a very interesting point. The notAliasCache is definitively
> > questionable. The question though is to find a better way to deal with
> it.
> > An option would be to not support alias (yes, I know, pretty drastic,
> but if
> > it allows us to speed up the delivery of a production ready server, I
> think
> > this would be a good tradeoff. Also note that once we have a solid base,
> we
> > can reintroduce alias handling in a future version).
> >
> >> This maps a Dn to whether the entry is an
> >> alias. If we followed the normal way of merging what is read from
> >> partitions with the txn log, we would have to a way of merging what is
> >> read from this notaliascache with the log. This is not very difficult
> >> but as we have quite a number of these caches, having a separate merge
> >> and update logic for each of them is a pain and error prone.
> >>
> >> Instead, what I thought was to use a single system wide read-write
> >> lock for these caches. Since they are read mostly, a txn might update
> >> its lock to exclusive when it needs to change one of these caches and
> >> might release it when it actually commits/aborts.
> >
> > We may start a new thread to discuss further those cache issues. We have
> > many caches, for may different thinsg, and many different approaches to
> deal
> > with the (MRU, EhCache, etc...). It's probably a good timing for
> clarifying
> > this item...
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
Hi,
a few points and questions:
*I am planning to use a common ID for all partitions. I checked Hbase
partition and it uses UUID as the ID. Is it ok to use this for all
partitions?
*It seems that I need to move txn and log package implementations to
core-shared because of the recent reorg. Coresession and nexus
implementation is moved there and interceptors might need these
packages.
* When preparing txn log edits, I will need to log Modification and
Attribute objects and keep them in memory for a while. Is it necessary
to clone these objects while doing the logging at partition nexus
level? In other words, will they be modified after reaching nexus
level. I am thinking no but I wanted to check because I looked at
changelog implementation and it does cloning.
regards,
Selcuk
On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> Hi !
>
> comments inline...
>
> On 11/1/11 12:19 PM, Selcuk AYA wrote:
>>
>> thanks for the feedback! Please see inline:
>
> <snip/>
>>
>>
>> I guess the xdbm-partition Maven module is now going to be pretty damn
>> thin
>> or non-existent. Was this module destroyed and if not what actually
>> remains?
>> I think some util classes for tree based implementations remain there.
>> Also there some avl related classes.
>
> We can review this project and eventually move the remaining classes
> elsewhere.
>
>>
>>>> The todo is the following:
>>>> *add changes to keep track of dn space changes.
>>>> *test txn manager services
>>>> *move the modification code path in AbstractBTreePartition in xdbm to
>>>> high up to core. Probably modifications should be done in
>>>> partitionNexus and partition nexus should handle preparing txn log
>>>> edits and wal them. The overall flow for modifications will be:
>>>> -DefaultDirectoryService:
>>>> -begintxn
>>>> -execute interceptor chain
>>>> - handle txn abort, or conflict.
>>>>
>>> Moving this into the PartitionNexus might not be a good idea but no
>>> problem
>>> for now we can move it later. Let me explain why:
>>> Eventually we're going to enable a root Partition with Partition nesting
>>> and
>>> so when this happens the PartitionNexus will just be another nestable
>>> Partition since these will have to handle routing based on DN to other
>>> partitions residing/nested under it.
>>> I see two possible locations for this functionality:
>>> (1) Let the InterceptorChain itself handle this since it can demarcate
>>> the
>>> start and end of calls into the chain with Txn begin and abort/commit
>>> calls.
>>> It does this by making calls against the TxnManager which I guess is the
>>> entire facade for the transaction subsystem.
>>> ---OR---
>>> (2) Handle Txn demarcation within the CoreSession. However this might not
>>> be
>>> optimal due to the need to handle additional logic which might be
>>> required
>>> for handling chain re-entry concerns.
>>> NOTE: I've not actually looked at the code after these major moves so my
>>> advice might not be very dependable. I will try to setup my environment
>>> to
>>> get a better idea of these matters.
>>> However for the time being do whatever actually makes this thing work.
>>> Let's
>>> follow an agile methodology. This thing is big. So let's get it working
>>> with
>>> solid test coverage then we can actually look at shuffling things around
>>> to
>>> optimal positions. Not saying what you've chosen is wrong ... it might
>>> just
>>> present the need for some additional refactoring when other features
>>> might
>>> need to be introduced.
>>> These are some of the biggest changes to the architecture to have taken
>>> place in years and you're doing a great job.
>>
>> You are right for txn demarcation. I wrote it wrong in the email.
>> Demarcation has to be done either at defaultcoresession or
>> defaultoperationmanager.
>
> OperationManager is probably the right place.
>
> For inner operations, I'm almost 100% sure they aren't modifiying anything
> (the only modification done as a inner operation has been removed a while
> ago : it was dealing with the addition of the ModifyTimeStamp and
> ModifiersName attribute, and it' snow done in the main operation). So we are
> safe if they are done in the current external transaction.
>
>>
>> as for where to handle the change logic, as you mentioned we need a
>> place where all interceptor chain routes end up for modification and
>> ideally we should handle modification logic above partitions using
>> master table and index interfaces so that we have a common place to
>> prepare and apply txn log edits. PartitionNexus seems to fit this
>> requirement for now. If we add another layer above PartitionNexus
>> which can get master and index table from below layers and work with
>> them, we should be able to move the change logic up there.
>>
>>>> *move xdbm-search to core as well. Making search transactional will
>>>> mostly be mostly mechanica after this point I think(hope). It should
>>>> just use the wrappers the txn manager provides for index, master and
>>>> cursors it gets from the partitions.
>>>>
>>>> *handle caches various interceptors keep. I am thinking of handling
>>>> this with a common read-write lock.
>>>>
>>> This was the latest issue for which I see some more threads. Will look at
>>> that as well.
>>
>> This is the issue I talked you about briefly. These are mostly admin
>> caches that change infrequently and that are read mostly. The
>> difficulty with them is that they are not always entry caches. They
>> might map Dns to some logical property of the entry. A simple and good
>> example is notaliascache (whether we really need this is open to
>> debate but we have it now).
>
> This is a very interesting point. The notAliasCache is definitively
> questionable. The question though is to find a better way to deal with it.
> An option would be to not support alias (yes, I know, pretty drastic, but if
> it allows us to speed up the delivery of a production ready server, I think
> this would be a good tradeoff. Also note that once we have a solid base, we
> can reintroduce alias handling in a future version).
>
>> This maps a Dn to whether the entry is an
>> alias. If we followed the normal way of merging what is read from
>> partitions with the txn log, we would have to a way of merging what is
>> read from this notaliascache with the log. This is not very difficult
>> but as we have quite a number of these caches, having a separate merge
>> and update logic for each of them is a pain and error prone.
>>
>> Instead, what I thought was to use a single system wide read-write
>> lock for these caches. Since they are read mostly, a txn might update
>> its lock to exclusive when it needs to change one of these caches and
>> might release it when it actually commits/aborts.
>
> We may start a new thread to discuss further those cache issues. We have
> many caches, for may different thinsg, and many different approaches to deal
> with the (MRU, EhCache, etc...). It's probably a good timing for clarifying
> this item...
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi !
comments inline...
On 11/1/11 12:19 PM, Selcuk AYA wrote:
> thanks for the feedback! Please see inline:
<snip/>
>
>
> I guess the xdbm-partition Maven module is now going to be pretty damn thin
> or non-existent. Was this module destroyed and if not what actually remains?
> I think some util classes for tree based implementations remain there.
> Also there some avl related classes.
We can review this project and eventually move the remaining classes
elsewhere.
>
>>> The todo is the following:
>>> *add changes to keep track of dn space changes.
>>> *test txn manager services
>>> *move the modification code path in AbstractBTreePartition in xdbm to
>>> high up to core. Probably modifications should be done in
>>> partitionNexus and partition nexus should handle preparing txn log
>>> edits and wal them. The overall flow for modifications will be:
>>> -DefaultDirectoryService:
>>> -begintxn
>>> -execute interceptor chain
>>> - handle txn abort, or conflict.
>>>
>> Moving this into the PartitionNexus might not be a good idea but no problem
>> for now we can move it later. Let me explain why:
>> Eventually we're going to enable a root Partition with Partition nesting and
>> so when this happens the PartitionNexus will just be another nestable
>> Partition since these will have to handle routing based on DN to other
>> partitions residing/nested under it.
>> I see two possible locations for this functionality:
>> (1) Let the InterceptorChain itself handle this since it can demarcate the
>> start and end of calls into the chain with Txn begin and abort/commit calls.
>> It does this by making calls against the TxnManager which I guess is the
>> entire facade for the transaction subsystem.
>> ---OR---
>> (2) Handle Txn demarcation within the CoreSession. However this might not be
>> optimal due to the need to handle additional logic which might be required
>> for handling chain re-entry concerns.
>> NOTE: I've not actually looked at the code after these major moves so my
>> advice might not be very dependable. I will try to setup my environment to
>> get a better idea of these matters.
>> However for the time being do whatever actually makes this thing work. Let's
>> follow an agile methodology. This thing is big. So let's get it working with
>> solid test coverage then we can actually look at shuffling things around to
>> optimal positions. Not saying what you've chosen is wrong ... it might just
>> present the need for some additional refactoring when other features might
>> need to be introduced.
>> These are some of the biggest changes to the architecture to have taken
>> place in years and you're doing a great job.
> You are right for txn demarcation. I wrote it wrong in the email.
> Demarcation has to be done either at defaultcoresession or
> defaultoperationmanager.
OperationManager is probably the right place.
For inner operations, I'm almost 100% sure they aren't modifiying
anything (the only modification done as a inner operation has been
removed a while ago : it was dealing with the addition of the
ModifyTimeStamp and ModifiersName attribute, and it' snow done in the
main operation). So we are safe if they are done in the current external
transaction.
>
> as for where to handle the change logic, as you mentioned we need a
> place where all interceptor chain routes end up for modification and
> ideally we should handle modification logic above partitions using
> master table and index interfaces so that we have a common place to
> prepare and apply txn log edits. PartitionNexus seems to fit this
> requirement for now. If we add another layer above PartitionNexus
> which can get master and index table from below layers and work with
> them, we should be able to move the change logic up there.
>
>>> *move xdbm-search to core as well. Making search transactional will
>>> mostly be mostly mechanica after this point I think(hope). It should
>>> just use the wrappers the txn manager provides for index, master and
>>> cursors it gets from the partitions.
>>>
>>> *handle caches various interceptors keep. I am thinking of handling
>>> this with a common read-write lock.
>>>
>> This was the latest issue for which I see some more threads. Will look at
>> that as well.
> This is the issue I talked you about briefly. These are mostly admin
> caches that change infrequently and that are read mostly. The
> difficulty with them is that they are not always entry caches. They
> might map Dns to some logical property of the entry. A simple and good
> example is notaliascache (whether we really need this is open to
> debate but we have it now).
This is a very interesting point. The notAliasCache is definitively
questionable. The question though is to find a better way to deal with
it. An option would be to not support alias (yes, I know, pretty
drastic, but if it allows us to speed up the delivery of a production
ready server, I think this would be a good tradeoff. Also note that once
we have a solid base, we can reintroduce alias handling in a future
version).
> This maps a Dn to whether the entry is an
> alias. If we followed the normal way of merging what is read from
> partitions with the txn log, we would have to a way of merging what is
> read from this notaliascache with the log. This is not very difficult
> but as we have quite a number of these caches, having a separate merge
> and update logic for each of them is a pain and error prone.
>
> Instead, what I thought was to use a single system wide read-write
> lock for these caches. Since they are read mostly, a txn might update
> its lock to exclusive when it needs to change one of these caches and
> might release it when it actually commits/aborts.
We may start a new thread to discuss further those cache issues. We have
many caches, for may different thinsg, and many different approaches to
deal with the (MRU, EhCache, etc...). It's probably a good timing for
clarifying this item...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
thanks for the feedback! Please see inline:
On Tue, Nov 1, 2011 at 5:35 AM, Alex Karasulu <ak...@apache.org> wrote:
> On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
>
> Excuse the late response. Better late than never right ...
> SNIP ...
>
>>
>> *Master table, index and index cursor wrappers: These provide
>> transactionally consistent views of entries and index data. They merge
>> what they read from their wrapped objects with the txn log manager. To
>> make this easier, txn manager keeps track of index changes for a
>> read/write txn in a summary form. One thing to note is that not all
>> information that wrappers expose is transactionall consistent. One
>> example is count methods. No need to %100 accurate on this. For these
>> cases, the wrapper just calls the wrapped object.
>>
>
> Makes perfect sense to not bother with the count methods: slight variations
> will not at all impact the optimizer's results.
>
>>
>> *For all the above, interfaces are defined in core-api and
>> implementations and internal interfaces are in core. Also moved,
>> index, indexcursor, table and related interfaces and helper classes to
>> core-api from xdbm-partition.
>>
>
> I guess the xdbm-partition Maven module is now going to be pretty damn thin
> or non-existent. Was this module destroyed and if not what actually remains?
I think some util classes for tree based implementations remain there.
Also there some avl related classes.
>
>>
>> The todo is the following:
>> *add changes to keep track of dn space changes.
>> *test txn manager services
>> *move the modification code path in AbstractBTreePartition in xdbm to
>> high up to core. Probably modifications should be done in
>> partitionNexus and partition nexus should handle preparing txn log
>> edits and wal them. The overall flow for modifications will be:
>> -DefaultDirectoryService:
>> -begintxn
>> -execute interceptor chain
>> - handle txn abort, or conflict.
>>
>
> Moving this into the PartitionNexus might not be a good idea but no problem
> for now we can move it later. Let me explain why:
> Eventually we're going to enable a root Partition with Partition nesting and
> so when this happens the PartitionNexus will just be another nestable
> Partition since these will have to handle routing based on DN to other
> partitions residing/nested under it.
> I see two possible locations for this functionality:
> (1) Let the InterceptorChain itself handle this since it can demarcate the
> start and end of calls into the chain with Txn begin and abort/commit calls.
> It does this by making calls against the TxnManager which I guess is the
> entire facade for the transaction subsystem.
> ---OR---
> (2) Handle Txn demarcation within the CoreSession. However this might not be
> optimal due to the need to handle additional logic which might be required
> for handling chain re-entry concerns.
> NOTE: I've not actually looked at the code after these major moves so my
> advice might not be very dependable. I will try to setup my environment to
> get a better idea of these matters.
> However for the time being do whatever actually makes this thing work. Let's
> follow an agile methodology. This thing is big. So let's get it working with
> solid test coverage then we can actually look at shuffling things around to
> optimal positions. Not saying what you've chosen is wrong ... it might just
> present the need for some additional refactoring when other features might
> need to be introduced.
> These are some of the biggest changes to the architecture to have taken
> place in years and you're doing a great job.
You are right for txn demarcation. I wrote it wrong in the email.
Demarcation has to be done either at defaultcoresession or
defaultoperationmanager.
as for where to handle the change logic, as you mentioned we need a
place where all interceptor chain routes end up for modification and
ideally we should handle modification logic above partitions using
master table and index interfaces so that we have a common place to
prepare and apply txn log edits. PartitionNexus seems to fit this
requirement for now. If we add another layer above PartitionNexus
which can get master and index table from below layers and work with
them, we should be able to move the change logic up there.
>>
>> *move xdbm-search to core as well. Making search transactional will
>> mostly be mostly mechanica after this point I think(hope). It should
>> just use the wrappers the txn manager provides for index, master and
>> cursors it gets from the partitions.
>>
>> *handle caches various interceptors keep. I am thinking of handling
>> this with a common read-write lock.
>>
>
> This was the latest issue for which I see some more threads. Will look at
> that as well.
This is the issue I talked you about briefly. These are mostly admin
caches that change infrequently and that are read mostly. The
difficulty with them is that they are not always entry caches. They
might map Dns to some logical property of the entry. A simple and good
example is notaliascache (whether we really need this is open to
debate but we have it now). This maps a Dn to whether the entry is an
alias. If we followed the normal way of merging what is read from
partitions with the txn log, we would have to a way of merging what is
read from this notaliascache with the log. This is not very difficult
but as we have quite a number of these caches, having a separate merge
and update logic for each of them is a pain and error prone.
Instead, what I thought was to use a single system wide read-write
lock for these caches. Since they are read mostly, a txn might update
its lock to exclusive when it needs to change one of these caches and
might release it when it actually commits/aborts.
> Again sorry about late responses.
> Regards,
> Alex
>
>>
>> regards
>> Selcuk
>>
>> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA <ay...@gmail.com> wrote:
>> > Hi all,
>> >
>> > Emmanuel wants to update the txns branch and I think it is a good time
>> > to give a good status update. Following is what I have been doing and
>> > the plan:
>> >
>> > *Log Manager: A general log manager that can be used by txns as well.
>> > Note that these logs are on disk. This is tested.
>> >
>> > *Txn Manager:Most of the services that should be provided by the txn
>> > manager is coded:
>> > begin, commit, abort, keeping
>> >
>
>
>
> --
> Best Regards,
> -- Alex
>