You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Selcuk AYA <ay...@gmail.com> on 2011/10/20 15:05:53 UTC
TXN work update
Hi all,
Emmanuel wants to update the txns branch and I think it is a good time
to give a good status update. Following is what I have been doing and
the plan:
*Log Manager: A general log manager that can be used by txns as well.
Note that these logs are on disk. This is tested.
*Txn Manager:Most of the services that should be provided by the txn
manager is coded:
begin, commit, abort, keeping
Re: TXN work update
Posted by Emmanuel Lécharny <el...@apache.org>.
On 10/24/11 8:40 PM, Selcuk AYA wrote:
> On Mon, Oct 24, 2011 at 6:26 PM, Emmanuel Lecharny<el...@gmail.com> wrote:
>> On 10/20/11 3:24 PM, Selcuk AYA wrote:
>>> Continuing the previous incomplete email:)
>>> Emmanuel wants to update the txns branch and I think it is a good time
>>> to give a good status update. Following is what I have been doing and
>>> the plan:
>>>
>>> *Log Manager: A general log manager that can be used by txns as well.
>>> Note that these logs are on disk. This is tested.
>> I reviewed the LogManager (and the associated classes) this week-end and
>> today. I committed a few fixes, no big errors. I have a few comments, not
>> all of them are mandatory, some of them are suggestions :
>>
>> o Rename the Log class to TransactionLog, to avoid any confusion with the
>> ChangeLog. Same for LogManager, LogFileManager, LogFlushManager.
> this log is not just for txns. İt is a general logging solution. In
> fact you can make the changelog use it instead of storing eveything in
> memory. Still if you have a better suggestion, I can change the names.
Ok, fine. I thought it wasn"'t generic, but looking at the code, you are
right. Actually, we already discussed with Julien Vermillard (MINA
project) about reusing this part in MINA.
>> o The methods in an Interface don't need to be declared as public : they
>> already are.
>> o Don't forget to apply the ADS formatter : 2 lines between each methods, a
>> newline before a return, etc…
> OK. I think I will use the formatter to format it one final time
> before check in after this time.
What you can do is to set Eclipse to format the code when you save the
files. It's incremental, you don't even have to think about it.
Same for headers, you can use the provided template that add them when
you create a new file. Very convenient.
>> o Don't add 'this.' when not necessary.
>> o Always use { and } when writing a if/for/while/do
> OK
>> o Try to avoid inner classes, except to hide the inner class behavior
> I sometimes use them to logically group data used together but
> strongly tied to a class. and I think it makes sense to use them in
> that way.
If they are really tied, yes, it makes sense to hide them. For instance,
the LogReader and writer are good candidate for inner classes : they
aren't likely to be used elswhere. But always be careful about creating
too many inner classes : it makes it difficult to find a class by its name.
>> o Don't declare an interface into another one
>> o LogFileManager should be moved to core-api
>> o LogScannerInterval should be moved to core-api
> the two above are internal interfaces that are used internally. I
> thought core-api has public interfaces.
Good point. This has to be discussed further.
>> o The LogFileRecords class might better be an interface ?
> fine as it is I think. This just groups data related to internal
> format of the log implementation. NOt every log implementation has to
> have them.
Ok.
>> o fields in classes must be private or protected
> OK
>> o Transaction should be moved to core-api
>> o TxnManagerInternal should be moved to core-api
> See comment for LogFileManager and LogScannerInternal
Ok.
FTR, I have made some few modifications in the code, but nothing
structural so far. I just try to get a grip on what you've done, in
order to be able to help if anyone has some issue with this part later.
If you ask Alex, he will tell you that code cleaning is my favorite
game. It is. I *like* that. Although my perception of what is 'clean
code' can be quite personal, and I may be wrong from time to time.
Overall, the code you committed is pretty slick, and I liked it !
Thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
On Mon, Oct 24, 2011 at 6:26 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 10/20/11 3:24 PM, Selcuk AYA wrote:
>>
>> Continuing the previous incomplete email:)
>> Emmanuel wants to update the txns branch and I think it is a good time
>> to give a good status update. Following is what I have been doing and
>> the plan:
>>
>> *Log Manager: A general log manager that can be used by txns as well.
>> Note that these logs are on disk. This is tested.
>
> I reviewed the LogManager (and the associated classes) this week-end and
> today. I committed a few fixes, no big errors. I have a few comments, not
> all of them are mandatory, some of them are suggestions :
>
> o Rename the Log class to TransactionLog, to avoid any confusion with the
> ChangeLog. Same for LogManager, LogFileManager, LogFlushManager.
this log is not just for txns. İt is a general logging solution. In
fact you can make the changelog use it instead of storing eveything in
memory. Still if you have a better suggestion, I can change the names.
> o The methods in an Interface don't need to be declared as public : they
> already are.
> o Don't forget to apply the ADS formatter : 2 lines between each methods, a
> newline before a return, etc…
OK. I think I will use the formatter to format it one final time
before check in after this time.
> o Don't add 'this.' when not necessary.
> o Always use { and } when writing a if/for/while/do
OK
> o Try to avoid inner classes, except to hide the inner class behavior
I sometimes use them to logically group data used together but
strongly tied to a class. and I think it makes sense to use them in
that way.
> o Don't declare an interface into another one
> o LogFileManager should be moved to core-api
> o LogScannerInterval should be moved to core-api
the two above are internal interfaces that are used internally. I
thought core-api has public interfaces.
> o The LogFileRecords class might better be an interface ?
fine as it is I think. This just groups data related to internal
format of the log implementation. NOt every log implementation has to
have them.
> o fields in classes must be private or protected
OK
> o Transaction should be moved to core-api
> o TxnManagerInternal should be moved to core-api
See comment for LogFileManager and LogScannerInternal
> o Arrays are always initialized to the default init value of their
> components (ie, 0x00 for bytes) : no need to initialize them
OK.
>
> Many of this points may be discussed further.
>
> Otherwise, I'll continue to explore the code base to understand how it
> works.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 10/20/11 3:24 PM, Selcuk AYA wrote:
> Continuing the previous incomplete email:)
> Emmanuel wants to update the txns branch and I think it is a good time
> to give a good status update. Following is what I have been doing and
> the plan:
>
> *Log Manager: A general log manager that can be used by txns as well.
> Note that these logs are on disk. This is tested.
I reviewed the LogManager (and the associated classes) this week-end and
today. I committed a few fixes, no big errors. I have a few comments,
not all of them are mandatory, some of them are suggestions :
o Rename the Log class to TransactionLog, to avoid any confusion with
the ChangeLog. Same for LogManager, LogFileManager, LogFlushManager.
o The methods in an Interface don't need to be declared as public : they
already are.
o Don't forget to apply the ADS formatter : 2 lines between each
methods, a newline before a return, etc…
o Don't add 'this.' when not necessary.
o Always use { and } when writing a if/for/while/do
o Try to avoid inner classes, except to hide the inner class behavior
o Don't declare an interface into another one
o LogFileManager should be moved to core-api
o LogScannerInterval should be moved to core-api
o The LogFileRecords class might better be an interface ?
o fields in classes must be private or protected
o Transaction should be moved to core-api
o TxnManagerInternal should be moved to core-api
o Arrays are always initialized to the default init value of their
components (ie, 0x00 for bytes) : no need to initialize them
Many of this points may be discussed further.
Otherwise, I'll continue to explore the code base to understand how it
works.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
On Tue, Nov 1, 2011 at 1:19 PM, Selcuk AYA <ay...@gmail.com> wrote:
> thanks for the feedback! Please see inline:
>
> On Tue, Nov 1, 2011 at 5:35 AM, Alex Karasulu <ak...@apache.org>
> wrote:
> > On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
> >
> > Excuse the late response. Better late than never right ...
> > SNIP ...
> >
> >>
> >> *Master table, index and index cursor wrappers: These provide
> >> transactionally consistent views of entries and index data. They merge
> >> what they read from their wrapped objects with the txn log manager. To
> >> make this easier, txn manager keeps track of index changes for a
> >> read/write txn in a summary form. One thing to note is that not all
> >> information that wrappers expose is transactionall consistent. One
> >> example is count methods. No need to %100 accurate on this. For these
> >> cases, the wrapper just calls the wrapped object.
> >>
> >
> > Makes perfect sense to not bother with the count methods: slight
> variations
> > will not at all impact the optimizer's results.
> >
> >>
> >> *For all the above, interfaces are defined in core-api and
> >> implementations and internal interfaces are in core. Also moved,
> >> index, indexcursor, table and related interfaces and helper classes to
> >> core-api from xdbm-partition.
> >>
> >
> > I guess the xdbm-partition Maven module is now going to be pretty damn
> thin
> > or non-existent. Was this module destroyed and if not what actually
> remains?
>
> I think some util classes for tree based implementations remain there.
> Also there some avl related classes.
>
> >
> >>
> >> The todo is the following:
> >> *add changes to keep track of dn space changes.
> >> *test txn manager services
> >> *move the modification code path in AbstractBTreePartition in xdbm to
> >> high up to core. Probably modifications should be done in
> >> partitionNexus and partition nexus should handle preparing txn log
> >> edits and wal them. The overall flow for modifications will be:
> >> -DefaultDirectoryService:
> >> -begintxn
> >> -execute interceptor chain
> >> - handle txn abort, or conflict.
> >>
> >
> > Moving this into the PartitionNexus might not be a good idea but no
> problem
> > for now we can move it later. Let me explain why:
> > Eventually we're going to enable a root Partition with Partition nesting
> and
> > so when this happens the PartitionNexus will just be another nestable
> > Partition since these will have to handle routing based on DN to other
> > partitions residing/nested under it.
> > I see two possible locations for this functionality:
> > (1) Let the InterceptorChain itself handle this since it can demarcate
> the
> > start and end of calls into the chain with Txn begin and abort/commit
> calls.
> > It does this by making calls against the TxnManager which I guess is the
> > entire facade for the transaction subsystem.
> > ---OR---
> > (2) Handle Txn demarcation within the CoreSession. However this might
> not be
> > optimal due to the need to handle additional logic which might be
> required
> > for handling chain re-entry concerns.
> > NOTE: I've not actually looked at the code after these major moves so my
> > advice might not be very dependable. I will try to setup my environment
> to
> > get a better idea of these matters.
> > However for the time being do whatever actually makes this thing work.
> Let's
> > follow an agile methodology. This thing is big. So let's get it working
> with
> > solid test coverage then we can actually look at shuffling things around
> to
> > optimal positions. Not saying what you've chosen is wrong ... it might
> just
> > present the need for some additional refactoring when other features
> might
> > need to be introduced.
> > These are some of the biggest changes to the architecture to have taken
> > place in years and you're doing a great job.
>
> You are right for txn demarcation. I wrote it wrong in the email.
> Demarcation has to be done either at defaultcoresession or
> defaultoperationmanager.
>
> as for where to handle the change logic, as you mentioned we need a
> place where all interceptor chain routes end up for modification and
> ideally we should handle modification logic above partitions using
> master table and index interfaces so that we have a common place to
> prepare and apply txn log edits. PartitionNexus seems to fit this
> requirement for now. If we add another layer above PartitionNexus
> which can get master and index table from below layers and work with
> them, we should be able to move the change logic up there.
>
> >>
> >> *move xdbm-search to core as well. Making search transactional will
> >> mostly be mostly mechanica after this point I think(hope). It should
> >> just use the wrappers the txn manager provides for index, master and
> >> cursors it gets from the partitions.
> >>
> >> *handle caches various interceptors keep. I am thinking of handling
> >> this with a common read-write lock.
> >>
> >
> > This was the latest issue for which I see some more threads. Will look at
> > that as well.
> This is the issue I talked you about briefly. These are mostly admin
> caches that change infrequently and that are read mostly. The
> difficulty with them is that they are not always entry caches. They
> might map Dns to some logical property of the entry. A simple and good
> example is notaliascache (whether we really need this is open to
> debate but we have it now). This maps a Dn to whether the entry is an
> alias. If we followed the normal way of merging what is read from
> partitions with the txn log, we would have to a way of merging what is
> read from this notaliascache with the log. This is not very difficult
> but as we have quite a number of these caches, having a separate merge
> and update logic for each of them is a pain and error prone.
>
> Instead, what I thought was to use a single system wide read-write
> lock for these caches. Since they are read mostly, a txn might update
> its lock to exclusive when it needs to change one of these caches and
> might release it when it actually commits/aborts.
>
>
Yep I remember this conversation very well. I think this is the best route
for us to take for the time being. Handling each cache separate takes more
effort and it might not pay off so let's not do unnecessary
write-ahead-coding :-D.
Who knows it might be the best way for us to proceed from all aspects
including performance. You just can't tell these things until you test the
server in a stress test environment. That's why we need a solid test
environment.
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/2/11 10:10 AM, Selcuk AYA wrote:
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level? In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
>> Once at the nexus level, the elements won't be modified.
>> The ChangeLog stores (currently) data in memory, and those data are kept for
>> a while, so cross transactions. This is the reason why we clone the entry
>> and Attributes.
> with transactions, the attributes and modifications will be kept in
> memory cross transactions as well. But my understanding( and the thing
> I wanted to verify) is that Attributes, their Values and Modification
> objects should not change any more once they are at the Nexus level as
> these objects created per modification request and they are ready to
> be applied to Entry objects.
Once in the nexus, the objects aren't modified.
About the changelog, you have to know that we keep them as long as we
need the elements, but as we keep all of them, we are memory bound
anyway. Also we don't need to keep the original entry in the changeLog,
as we only have to apply the revert operation. The original entry is
just needed to compute the reverse entry.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
On Wed, Nov 2, 2011 at 10:52 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/2/11 9:12 AM, Selcuk AYA wrote:
>>
>> Hi,
>> a few points and questions:
>> *I am planning to use a common ID for all partitions. I checked Hbase
>> partition and it uses UUID as the ID. Is it ok to use this for all
>> partitions?
>
> Yes. This should have been done a long time ago.
>>
>> *It seems that I need to move txn and log package implementations to
>> core-shared because of the recent reorg. Coresession and nexus
>> implementation is moved there and interceptors might need these
>> packages.
>
> Yep. We haven't yet merged core-shared with core, but atm, core-shared is
> the place to put new classes.
>>
>> * When preparing txn log edits, I will need to log Modification and
>> Attribute objects and keep them in memory for a while. Is it necessary
>> to clone these objects while doing the logging at partition nexus
>> level? In other words, will they be modified after reaching nexus
>> level. I am thinking no but I wanted to check because I looked at
>> changelog implementation and it does cloning.
>
> Once at the nexus level, the elements won't be modified.
> The ChangeLog stores (currently) data in memory, and those data are kept for
> a while, so cross transactions. This is the reason why we clone the entry
> and Attributes.
with transactions, the attributes and modifications will be kept in
memory cross transactions as well. But my understanding( and the thing
I wanted to verify) is that Attributes, their Values and Modification
objects should not change any more once they are at the Nexus level as
these objects created per modification request and they are ready to
be applied to Entry objects.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/2/11 9:12 AM, Selcuk AYA wrote:
> Hi,
> a few points and questions:
> *I am planning to use a common ID for all partitions. I checked Hbase
> partition and it uses UUID as the ID. Is it ok to use this for all
> partitions?
Yes. This should have been done a long time ago.
>
> *It seems that I need to move txn and log package implementations to
> core-shared because of the recent reorg. Coresession and nexus
> implementation is moved there and interceptors might need these
> packages.
Yep. We haven't yet merged core-shared with core, but atm, core-shared
is the place to put new classes.
>
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level? In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
Once at the nexus level, the elements won't be modified.
The ChangeLog stores (currently) data in memory, and those data are kept
for a while, so cross transactions. This is the reason why we clone the
entry and Attributes.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/14/11 9:09 AM, Selcuk AYA wrote:
> hi,
Hi !
> i am currently working on changing modification layer to use txn
> services and have a few questions about dropaliasindices:
>
> *is it true that an entry has object class ALIAS_OC iff it contains
> attribute ALIASED_OBJECT_NAME_AT.
The Alias ObjectClass has one mandatory AttributeType, aliasedObjectName :
objectclass ( 2.5.6.1
NAME 'alias'
DESC 'RFC2256: an alias'
SUP top
STRUCTURAL
MUST aliasedObjectName
)
>
> *where can I check the ldap definitions of atttributes given in
> SchemaConstants. I wanted check if ALIASED_OBJECT_NAME_AT is single
> valued.
The aliasedObjectName is single valued :
attributetype ( 2.5.4.1
NAME ( 'aliasedObjectName' 'aliasedEntryName' )
DESC 'RFC2256: name of aliased object'
EQUALITY distinguishedNameMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.12
SINGLE-VALUE
)
The easiest way to get those information is not to google them, but to
use the schema editor in Apache Directory Studio :)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
hi,
i am currently working on changing modification layer to use txn
services and have a few questions about dropaliasindices:
*is it true that an entry has object class ALIAS_OC iff it contains
attribute ALIASED_OBJECT_NAME_AT.
*where can I check the ldap definitions of atttributes given in
SchemaConstants. I wanted check if ALIASED_OBJECT_NAME_AT is single
valued.
thanks
Selcuk
On Thu, Nov 10, 2011 at 10:51 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/10/11 7:56 AM, Selcuk AYA wrote:
>>
>> Hi,
>> I have made the changes to use UUID for ID. I have also removed some
>> unnecessary type parameters. In particular, master table is not
>> generic anymore. For index and index cursors, only value type is a
>> generic parameter. Search engine and partitions and tests are changes
>> accordingly. Note that counter maintained for ID generation in Btree
>> partitions are not necessary anymore. Also, admin table for Jdbm
>> parition is removed. One thing I noticed in the tests were implicit
>> assumption of master table and indices with the key ID having the same
>> order as the insertion order of entries. This assumption is not valid
>> anymore as UUID generation is random.
>
> Great job !
>>
>> There is one outstanding issue. Attribute types for system indices
>> having UUID as the key has integer comparator as their ordering rule
>> in the LDAP attribute type definition. This should be changed to use
>> UUID ldap comparator. This is not done yet as I dont know how to do
>> it. Instead, we check for specific attribute types during index
>> initializations and use UUID comparator if necessary.
>
> I can handle that.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/10/11 7:56 AM, Selcuk AYA wrote:
> Hi,
> I have made the changes to use UUID for ID. I have also removed some
> unnecessary type parameters. In particular, master table is not
> generic anymore. For index and index cursors, only value type is a
> generic parameter. Search engine and partitions and tests are changes
> accordingly. Note that counter maintained for ID generation in Btree
> partitions are not necessary anymore. Also, admin table for Jdbm
> parition is removed. One thing I noticed in the tests were implicit
> assumption of master table and indices with the key ID having the same
> order as the insertion order of entries. This assumption is not valid
> anymore as UUID generation is random.
Great job !
>
> There is one outstanding issue. Attribute types for system indices
> having UUID as the key has integer comparator as their ordering rule
> in the LDAP attribute type definition. This should be changed to use
> UUID ldap comparator. This is not done yet as I dont know how to do
> it. Instead, we check for specific attribute types during index
> initializations and use UUID comparator if necessary.
I can handle that.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
Hi,
I have made the changes to use UUID for ID. I have also removed some
unnecessary type parameters. In particular, master table is not
generic anymore. For index and index cursors, only value type is a
generic parameter. Search engine and partitions and tests are changes
accordingly. Note that counter maintained for ID generation in Btree
partitions are not necessary anymore. Also, admin table for Jdbm
parition is removed. One thing I noticed in the tests were implicit
assumption of master table and indices with the key ID having the same
order as the insertion order of entries. This assumption is not valid
anymore as UUID generation is random.
There is one outstanding issue. Attribute types for system indices
having UUID as the key has integer comparator as their ordering rule
in the LDAP attribute type definition. This should be changed to use
UUID ldap comparator. This is not done yet as I dont know how to do
it. Instead, we check for specific attribute types during index
initializations and use UUID comparator if necessary.
Currently all tests are passing.
regards,
Selcuk
On Fri, Nov 4, 2011 at 9:02 PM, Stefan Seelmann <se...@apache.org> wrote:
> On Fri, Nov 4, 2011 at 9:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
>> I discussed using generics for ID with Emmanuel a little more. The
>> related change is mostly mechanical but it is a big change so I would
>> like to see if people have any opinion. Remember that alongside with
>> transaction changes, we are going towards a generic transactional
>> modification layer on top of partitions and a transactional search
>> engine that can work with any partition. Default search engine
>> achieves what we want in this sense but it works with Store interface
>> rather than Partition interface. Default Search Engine is designed to
>> work with generic ID. Store interface also uses generic ID. Now, if we
>> want to move DefaultSearchEngine to core and want it to work with
>> Partition interfaces, we have to options:
>> 1)Make partition interface parametrized too. Change it to
>> Partition<ID> and make it expose master tables and indices using
>> generic ID. DefaultSearch engine can currently work with such an
>> interface. We will also make the transactional modification layer use
>> this parametrized interface.We will also need to implement partitions
>> using the generic ID type rather than Long or UUID
>>
>> 2) Use UUID all over rather than generics. Basically get rid of
>> generic ID type. Change search engine to use UUID as well.
>
> +1 to the 2nd option.
>
> I introduced the generid <ID> two years ago [1] while I as was
> implementing the HBase partition prototype. Prior to that the ID was a
> Long which didn't work well with HBase. Now that we all agree that
> UUID should be used I see no reason why the generic ID should be kept.
>
> Kind Regards,
> Stefan
>
> [1] https://issues.apache.org/jira/browse/DIRSERVER-1472
>
Re: TXN work update
Posted by Stefan Seelmann <se...@apache.org>.
On Fri, Nov 4, 2011 at 9:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
+1 to the 2nd option.
I introduced the generid <ID> two years ago [1] while I as was
implementing the HBase partition prototype. Prior to that the ID was a
Long which didn't work well with HBase. Now that we all agree that
UUID should be used I see no reason why the generic ID should be kept.
Kind Regards,
Stefan
[1] https://issues.apache.org/jira/browse/DIRSERVER-1472
Re: TXN work update
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 4 nov. 2011, at 10:24, Alex Karasulu wrote:
> I vote for the second option and to get rid of Generics on these interfaces. Why?
>
> (1) We need to go to UUIDs since we will soon be living in a replicated world and there's no point to a partition specific id.
> (2) We can still do (1) with a Long but why bother?
> (3) Using a UUID as the PK for entries gets rid of the need for the UUID index since the master automatically becomes the UUID index.
> (4) Generics have made everything way too complex in this region of the code and I do not foresee a need to use anything else at this point in time. If the need arises we can consider it after everything is working ... a simple maneuver/re-factoring.
+1
Although I like Generics a lot, it makes probably more sense to avoid them here in this particular context.
Regards,
Pierre-Arnaud
>
> Regards,
> Alex
>
> On Fri, Nov 4, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
>
>
> First option has the flexibility of changing ID to a different type
> but I am not sure it makes sense to change the ID from release to
> release. I am inclined to go with the second option at the moment.
>
> regards,
> Selcuk
>
> On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> > On 11/3/11 4:44 PM, Selcuk AYA wrote:
> >>
> >> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
> >> wrote:
> >>>
> >>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
> >>>>
> >>>> Hi,
> >>>> a few points and questions:
> >>>> *I am planning to use a common ID for all partitions. I checked Hbase
> >>>> partition and it uses UUID as the ID. Is it ok to use this for all
> >>>> partitions?
> >>>>
> >>> This great and something we wanted to do for a very long time as Emmanuel
> >>> stated.
> >>
> >> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> >> am making partitions expose their master tabe and indices. For making
> >> all partitions use UUID:
> >> * I can change partitions to work with UUID directly and make them
> >> return master tables as MasterTable<Entry,UUID> and index tables in a
> >> similar way
> >> *or I can paratmetrize partition interface like Partition<ID> . They
> >> can return master table as<MasterTable,ID> and index tables in a
> >> similar way. This is looks to me the right way but it eventually leads
> >> to more changes.
> >>
> >> which one do you think is better?
> >
> > Let's not use generic when it's not needed...
> >
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
>
>
> --
> Best Regards,
> -- Alex
>
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
I vote for the second option and to get rid of Generics on these
interfaces. Why?
(1) We need to go to UUIDs since we will soon be living in a replicated
world and there's no point to a partition specific id.
(2) We can still do (1) with a Long but why bother?
(3) Using a UUID as the PK for entries gets rid of the need for the UUID
index since the master automatically becomes the UUID index.
(4) Generics have made everything way too complex in this region of the
code and I do not foresee a need to use anything else at this point in
time. If the need arises we can consider it after everything is working ...
a simple maneuver/re-factoring.
Regards,
Alex
On Fri, Nov 4, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> I discussed using generics for ID with Emmanuel a little more. The
> related change is mostly mechanical but it is a big change so I would
> like to see if people have any opinion. Remember that alongside with
> transaction changes, we are going towards a generic transactional
> modification layer on top of partitions and a transactional search
> engine that can work with any partition. Default search engine
> achieves what we want in this sense but it works with Store interface
> rather than Partition interface. Default Search Engine is designed to
> work with generic ID. Store interface also uses generic ID. Now, if we
> want to move DefaultSearchEngine to core and want it to work with
> Partition interfaces, we have to options:
> 1)Make partition interface parametrized too. Change it to
> Partition<ID> and make it expose master tables and indices using
> generic ID. DefaultSearch engine can currently work with such an
> interface. We will also make the transactional modification layer use
> this parametrized interface.We will also need to implement partitions
> using the generic ID type rather than Long or UUID
>
> 2) Use UUID all over rather than generics. Basically get rid of
> generic ID type. Change search engine to use UUID as well.
>
>
> First option has the flexibility of changing ID to a different type
> but I am not sure it makes sense to change the ID from release to
> release. I am inclined to go with the second option at the moment.
>
> regards,
> Selcuk
>
> On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> > On 11/3/11 4:44 PM, Selcuk AYA wrote:
> >>
> >> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
> >> wrote:
> >>>
> >>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>> a few points and questions:
> >>>> *I am planning to use a common ID for all partitions. I checked Hbase
> >>>> partition and it uses UUID as the ID. Is it ok to use this for all
> >>>> partitions?
> >>>>
> >>> This great and something we wanted to do for a very long time as
> Emmanuel
> >>> stated.
> >>
> >> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> >> am making partitions expose their master tabe and indices. For making
> >> all partitions use UUID:
> >> * I can change partitions to work with UUID directly and make them
> >> return master tables as MasterTable<Entry,UUID> and index tables in a
> >> similar way
> >> *or I can paratmetrize partition interface like Partition<ID> . They
> >> can return master table as<MasterTable,ID> and index tables in a
> >> similar way. This is looks to me the right way but it eventually leads
> >> to more changes.
> >>
> >> which one do you think is better?
> >
> > Let's not use generic when it's not needed...
> >
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
I discussed using generics for ID with Emmanuel a little more. The
related change is mostly mechanical but it is a big change so I would
like to see if people have any opinion. Remember that alongside with
transaction changes, we are going towards a generic transactional
modification layer on top of partitions and a transactional search
engine that can work with any partition. Default search engine
achieves what we want in this sense but it works with Store interface
rather than Partition interface. Default Search Engine is designed to
work with generic ID. Store interface also uses generic ID. Now, if we
want to move DefaultSearchEngine to core and want it to work with
Partition interfaces, we have to options:
1)Make partition interface parametrized too. Change it to
Partition<ID> and make it expose master tables and indices using
generic ID. DefaultSearch engine can currently work with such an
interface. We will also make the transactional modification layer use
this parametrized interface.We will also need to implement partitions
using the generic ID type rather than Long or UUID
2) Use UUID all over rather than generics. Basically get rid of
generic ID type. Change search engine to use UUID as well.
First option has the flexibility of changing ID to a different type
but I am not sure it makes sense to change the ID from release to
release. I am inclined to go with the second option at the moment.
regards,
Selcuk
On Thu, Nov 3, 2011 at 5:52 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 11/3/11 4:44 PM, Selcuk AYA wrote:
>>
>> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org>
>> wrote:
>>>
>>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> a few points and questions:
>>>> *I am planning to use a common ID for all partitions. I checked Hbase
>>>> partition and it uses UUID as the ID. Is it ok to use this for all
>>>> partitions?
>>>>
>>> This great and something we wanted to do for a very long time as Emmanuel
>>> stated.
>>
>> I am working on JDBM and AVL partitions to make them usee UUID. Also I
>> am making partitions expose their master tabe and indices. For making
>> all partitions use UUID:
>> * I can change partitions to work with UUID directly and make them
>> return master tables as MasterTable<Entry,UUID> and index tables in a
>> similar way
>> *or I can paratmetrize partition interface like Partition<ID> . They
>> can return master table as<MasterTable,ID> and index tables in a
>> similar way. This is looks to me the right way but it eventually leads
>> to more changes.
>>
>> which one do you think is better?
>
> Let's not use generic when it's not needed...
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 11/3/11 4:44 PM, Selcuk AYA wrote:
> On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu<ak...@apache.org> wrote:
>> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA<ay...@gmail.com> wrote:
>>> Hi,
>>> a few points and questions:
>>> *I am planning to use a common ID for all partitions. I checked Hbase
>>> partition and it uses UUID as the ID. Is it ok to use this for all
>>> partitions?
>>>
>> This great and something we wanted to do for a very long time as Emmanuel
>> stated.
> I am working on JDBM and AVL partitions to make them usee UUID. Also I
> am making partitions expose their master tabe and indices. For making
> all partitions use UUID:
> * I can change partitions to work with UUID directly and make them
> return master tables as MasterTable<Entry,UUID> and index tables in a
> similar way
> *or I can paratmetrize partition interface like Partition<ID> . They
> can return master table as<MasterTable,ID> and index tables in a
> similar way. This is looks to me the right way but it eventually leads
> to more changes.
>
> which one do you think is better?
Let's not use generic when it's not needed...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
On Wed, Nov 2, 2011 at 12:15 PM, Alex Karasulu <ak...@apache.org> wrote:
> On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
>>
>> Hi,
>> a few points and questions:
>> *I am planning to use a common ID for all partitions. I checked Hbase
>> partition and it uses UUID as the ID. Is it ok to use this for all
>> partitions?
>>
>
> This great and something we wanted to do for a very long time as Emmanuel
> stated.
I am working on JDBM and AVL partitions to make them usee UUID. Also I
am making partitions expose their master tabe and indices. For making
all partitions use UUID:
* I can change partitions to work with UUID directly and make them
return master tables as MasterTable<Entry,UUID> and index tables in a
similar way
*or I can paratmetrize partition interface like Partition<ID> . They
can return master table as <MasterTable,ID> and index tables in a
similar way. This is looks to me the right way but it eventually leads
to more changes.
which one do you think is better?
>
>>
>> *It seems that I need to move txn and log package implementations to
>> core-shared because of the recent reorg. Coresession and nexus
>> implementation is moved there and interceptors might need these
>> packages.
>>
>
> +1
>
>>
>> * When preparing txn log edits, I will need to log Modification and
>> Attribute objects and keep them in memory for a while. Is it necessary
>> to clone these objects while doing the logging at partition nexus
>> level?
>
> Don't think so.
>
>>
>> In other words, will they be modified after reaching nexus
>> level. I am thinking no but I wanted to check because I looked at
>> changelog implementation and it does cloning.
>>
>
> Thanks,
> Alex
>
>>
>> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com>
>> wrote:
>> > Hi !
>> >
>> > comments inline...
>> >
>> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
>> >>
>> >> thanks for the feedback! Please see inline:
>> >
>> > <snip/>
>> >>
>> >>
>> >> I guess the xdbm-partition Maven module is now going to be pretty damn
>> >> thin
>> >> or non-existent. Was this module destroyed and if not what actually
>> >> remains?
>> >> I think some util classes for tree based implementations remain there.
>> >> Also there some avl related classes.
>> >
>> > We can review this project and eventually move the remaining classes
>> > elsewhere.
>> >
>> >>
>> >>>> The todo is the following:
>> >>>> *add changes to keep track of dn space changes.
>> >>>> *test txn manager services
>> >>>> *move the modification code path in AbstractBTreePartition in xdbm to
>> >>>> high up to core. Probably modifications should be done in
>> >>>> partitionNexus and partition nexus should handle preparing txn log
>> >>>> edits and wal them. The overall flow for modifications will be:
>> >>>> -DefaultDirectoryService:
>> >>>> -begintxn
>> >>>> -execute interceptor chain
>> >>>> - handle txn abort, or conflict.
>> >>>>
>> >>> Moving this into the PartitionNexus might not be a good idea but no
>> >>> problem
>> >>> for now we can move it later. Let me explain why:
>> >>> Eventually we're going to enable a root Partition with Partition
>> >>> nesting
>> >>> and
>> >>> so when this happens the PartitionNexus will just be another nestable
>> >>> Partition since these will have to handle routing based on DN to other
>> >>> partitions residing/nested under it.
>> >>> I see two possible locations for this functionality:
>> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
>> >>> the
>> >>> start and end of calls into the chain with Txn begin and abort/commit
>> >>> calls.
>> >>> It does this by making calls against the TxnManager which I guess is
>> >>> the
>> >>> entire facade for the transaction subsystem.
>> >>> ---OR---
>> >>> (2) Handle Txn demarcation within the CoreSession. However this might
>> >>> not
>> >>> be
>> >>> optimal due to the need to handle additional logic which might be
>> >>> required
>> >>> for handling chain re-entry concerns.
>> >>> NOTE: I've not actually looked at the code after these major moves so
>> >>> my
>> >>> advice might not be very dependable. I will try to setup my
>> >>> environment
>> >>> to
>> >>> get a better idea of these matters.
>> >>> However for the time being do whatever actually makes this thing work.
>> >>> Let's
>> >>> follow an agile methodology. This thing is big. So let's get it
>> >>> working
>> >>> with
>> >>> solid test coverage then we can actually look at shuffling things
>> >>> around
>> >>> to
>> >>> optimal positions. Not saying what you've chosen is wrong ... it might
>> >>> just
>> >>> present the need for some additional refactoring when other features
>> >>> might
>> >>> need to be introduced.
>> >>> These are some of the biggest changes to the architecture to have
>> >>> taken
>> >>> place in years and you're doing a great job.
>> >>
>> >> You are right for txn demarcation. I wrote it wrong in the email.
>> >> Demarcation has to be done either at defaultcoresession or
>> >> defaultoperationmanager.
>> >
>> > OperationManager is probably the right place.
>> >
>> > For inner operations, I'm almost 100% sure they aren't modifiying
>> > anything
>> > (the only modification done as a inner operation has been removed a
>> > while
>> > ago : it was dealing with the addition of the ModifyTimeStamp and
>> > ModifiersName attribute, and it' snow done in the main operation). So we
>> > are
>> > safe if they are done in the current external transaction.
>> >
>> >>
>> >> as for where to handle the change logic, as you mentioned we need a
>> >> place where all interceptor chain routes end up for modification and
>> >> ideally we should handle modification logic above partitions using
>> >> master table and index interfaces so that we have a common place to
>> >> prepare and apply txn log edits. PartitionNexus seems to fit this
>> >> requirement for now. If we add another layer above PartitionNexus
>> >> which can get master and index table from below layers and work with
>> >> them, we should be able to move the change logic up there.
>> >>
>> >>>> *move xdbm-search to core as well. Making search transactional will
>> >>>> mostly be mostly mechanica after this point I think(hope). It should
>> >>>> just use the wrappers the txn manager provides for index, master and
>> >>>> cursors it gets from the partitions.
>> >>>>
>> >>>> *handle caches various interceptors keep. I am thinking of handling
>> >>>> this with a common read-write lock.
>> >>>>
>> >>> This was the latest issue for which I see some more threads. Will look
>> >>> at
>> >>> that as well.
>> >>
>> >> This is the issue I talked you about briefly. These are mostly admin
>> >> caches that change infrequently and that are read mostly. The
>> >> difficulty with them is that they are not always entry caches. They
>> >> might map Dns to some logical property of the entry. A simple and good
>> >> example is notaliascache (whether we really need this is open to
>> >> debate but we have it now).
>> >
>> > This is a very interesting point. The notAliasCache is definitively
>> > questionable. The question though is to find a better way to deal with
>> > it.
>> > An option would be to not support alias (yes, I know, pretty drastic,
>> > but if
>> > it allows us to speed up the delivery of a production ready server, I
>> > think
>> > this would be a good tradeoff. Also note that once we have a solid base,
>> > we
>> > can reintroduce alias handling in a future version).
>> >
>> >> This maps a Dn to whether the entry is an
>> >> alias. If we followed the normal way of merging what is read from
>> >> partitions with the txn log, we would have to a way of merging what is
>> >> read from this notaliascache with the log. This is not very difficult
>> >> but as we have quite a number of these caches, having a separate merge
>> >> and update logic for each of them is a pain and error prone.
>> >>
>> >> Instead, what I thought was to use a single system wide read-write
>> >> lock for these caches. Since they are read mostly, a txn might update
>> >> its lock to exclusive when it needs to change one of these caches and
>> >> might release it when it actually commits/aborts.
>> >
>> > We may start a new thread to discuss further those cache issues. We have
>> > many caches, for may different thinsg, and many different approaches to
>> > deal
>> > with the (MRU, EhCache, etc...). It's probably a good timing for
>> > clarifying
>> > this item...
>> >
>> > --
>> > Regards,
>> > Cordialement,
>> > Emmanuel Lécharny
>> > www.iktek.com
>> >
>> >
>
>
>
> --
> Best Regards,
> -- Alex
>
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ay...@gmail.com> wrote:
> Hi,
> a few points and questions:
> *I am planning to use a common ID for all partitions. I checked Hbase
> partition and it uses UUID as the ID. Is it ok to use this for all
> partitions?
>
>
This great and something we wanted to do for a very long time as Emmanuel
stated.
> *It seems that I need to move txn and log package implementations to
> core-shared because of the recent reorg. Coresession and nexus
> implementation is moved there and interceptors might need these
> packages.
>
>
+1
> * When preparing txn log edits, I will need to log Modification and
> Attribute objects and keep them in memory for a while. Is it necessary
> to clone these objects while doing the logging at partition nexus
> level?
Don't think so.
> In other words, will they be modified after reaching nexus
> level. I am thinking no but I wanted to check because I looked at
> changelog implementation and it does cloning.
>
>
Thanks,
Alex
>
> On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> > Hi !
> >
> > comments inline...
> >
> > On 11/1/11 12:19 PM, Selcuk AYA wrote:
> >>
> >> thanks for the feedback! Please see inline:
> >
> > <snip/>
> >>
> >>
> >> I guess the xdbm-partition Maven module is now going to be pretty damn
> >> thin
> >> or non-existent. Was this module destroyed and if not what actually
> >> remains?
> >> I think some util classes for tree based implementations remain there.
> >> Also there some avl related classes.
> >
> > We can review this project and eventually move the remaining classes
> > elsewhere.
> >
> >>
> >>>> The todo is the following:
> >>>> *add changes to keep track of dn space changes.
> >>>> *test txn manager services
> >>>> *move the modification code path in AbstractBTreePartition in xdbm to
> >>>> high up to core. Probably modifications should be done in
> >>>> partitionNexus and partition nexus should handle preparing txn log
> >>>> edits and wal them. The overall flow for modifications will be:
> >>>> -DefaultDirectoryService:
> >>>> -begintxn
> >>>> -execute interceptor chain
> >>>> - handle txn abort, or conflict.
> >>>>
> >>> Moving this into the PartitionNexus might not be a good idea but no
> >>> problem
> >>> for now we can move it later. Let me explain why:
> >>> Eventually we're going to enable a root Partition with Partition
> nesting
> >>> and
> >>> so when this happens the PartitionNexus will just be another nestable
> >>> Partition since these will have to handle routing based on DN to other
> >>> partitions residing/nested under it.
> >>> I see two possible locations for this functionality:
> >>> (1) Let the InterceptorChain itself handle this since it can demarcate
> >>> the
> >>> start and end of calls into the chain with Txn begin and abort/commit
> >>> calls.
> >>> It does this by making calls against the TxnManager which I guess is
> the
> >>> entire facade for the transaction subsystem.
> >>> ---OR---
> >>> (2) Handle Txn demarcation within the CoreSession. However this might
> not
> >>> be
> >>> optimal due to the need to handle additional logic which might be
> >>> required
> >>> for handling chain re-entry concerns.
> >>> NOTE: I've not actually looked at the code after these major moves so
> my
> >>> advice might not be very dependable. I will try to setup my environment
> >>> to
> >>> get a better idea of these matters.
> >>> However for the time being do whatever actually makes this thing work.
> >>> Let's
> >>> follow an agile methodology. This thing is big. So let's get it working
> >>> with
> >>> solid test coverage then we can actually look at shuffling things
> around
> >>> to
> >>> optimal positions. Not saying what you've chosen is wrong ... it might
> >>> just
> >>> present the need for some additional refactoring when other features
> >>> might
> >>> need to be introduced.
> >>> These are some of the biggest changes to the architecture to have taken
> >>> place in years and you're doing a great job.
> >>
> >> You are right for txn demarcation. I wrote it wrong in the email.
> >> Demarcation has to be done either at defaultcoresession or
> >> defaultoperationmanager.
> >
> > OperationManager is probably the right place.
> >
> > For inner operations, I'm almost 100% sure they aren't modifiying
> anything
> > (the only modification done as a inner operation has been removed a while
> > ago : it was dealing with the addition of the ModifyTimeStamp and
> > ModifiersName attribute, and it' snow done in the main operation). So we
> are
> > safe if they are done in the current external transaction.
> >
> >>
> >> as for where to handle the change logic, as you mentioned we need a
> >> place where all interceptor chain routes end up for modification and
> >> ideally we should handle modification logic above partitions using
> >> master table and index interfaces so that we have a common place to
> >> prepare and apply txn log edits. PartitionNexus seems to fit this
> >> requirement for now. If we add another layer above PartitionNexus
> >> which can get master and index table from below layers and work with
> >> them, we should be able to move the change logic up there.
> >>
> >>>> *move xdbm-search to core as well. Making search transactional will
> >>>> mostly be mostly mechanica after this point I think(hope). It should
> >>>> just use the wrappers the txn manager provides for index, master and
> >>>> cursors it gets from the partitions.
> >>>>
> >>>> *handle caches various interceptors keep. I am thinking of handling
> >>>> this with a common read-write lock.
> >>>>
> >>> This was the latest issue for which I see some more threads. Will look
> at
> >>> that as well.
> >>
> >> This is the issue I talked you about briefly. These are mostly admin
> >> caches that change infrequently and that are read mostly. The
> >> difficulty with them is that they are not always entry caches. They
> >> might map Dns to some logical property of the entry. A simple and good
> >> example is notaliascache (whether we really need this is open to
> >> debate but we have it now).
> >
> > This is a very interesting point. The notAliasCache is definitively
> > questionable. The question though is to find a better way to deal with
> it.
> > An option would be to not support alias (yes, I know, pretty drastic,
> but if
> > it allows us to speed up the delivery of a production ready server, I
> think
> > this would be a good tradeoff. Also note that once we have a solid base,
> we
> > can reintroduce alias handling in a future version).
> >
> >> This maps a Dn to whether the entry is an
> >> alias. If we followed the normal way of merging what is read from
> >> partitions with the txn log, we would have to a way of merging what is
> >> read from this notaliascache with the log. This is not very difficult
> >> but as we have quite a number of these caches, having a separate merge
> >> and update logic for each of them is a pain and error prone.
> >>
> >> Instead, what I thought was to use a single system wide read-write
> >> lock for these caches. Since they are read mostly, a txn might update
> >> its lock to exclusive when it needs to change one of these caches and
> >> might release it when it actually commits/aborts.
> >
> > We may start a new thread to discuss further those cache issues. We have
> > many caches, for may different thinsg, and many different approaches to
> deal
> > with the (MRU, EhCache, etc...). It's probably a good timing for
> clarifying
> > this item...
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
Hi,
a few points and questions:
*I am planning to use a common ID for all partitions. I checked Hbase
partition and it uses UUID as the ID. Is it ok to use this for all
partitions?
*It seems that I need to move txn and log package implementations to
core-shared because of the recent reorg. Coresession and nexus
implementation is moved there and interceptors might need these
packages.
* When preparing txn log edits, I will need to log Modification and
Attribute objects and keep them in memory for a while. Is it necessary
to clone these objects while doing the logging at partition nexus
level? In other words, will they be modified after reaching nexus
level. I am thinking no but I wanted to check because I looked at
changelog implementation and it does cloning.
regards,
Selcuk
On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> Hi !
>
> comments inline...
>
> On 11/1/11 12:19 PM, Selcuk AYA wrote:
>>
>> thanks for the feedback! Please see inline:
>
> <snip/>
>>
>>
>> I guess the xdbm-partition Maven module is now going to be pretty damn
>> thin
>> or non-existent. Was this module destroyed and if not what actually
>> remains?
>> I think some util classes for tree based implementations remain there.
>> Also there some avl related classes.
>
> We can review this project and eventually move the remaining classes
> elsewhere.
>
>>
>>>> The todo is the following:
>>>> *add changes to keep track of dn space changes.
>>>> *test txn manager services
>>>> *move the modification code path in AbstractBTreePartition in xdbm to
>>>> high up to core. Probably modifications should be done in
>>>> partitionNexus and partition nexus should handle preparing txn log
>>>> edits and wal them. The overall flow for modifications will be:
>>>> -DefaultDirectoryService:
>>>> -begintxn
>>>> -execute interceptor chain
>>>> - handle txn abort, or conflict.
>>>>
>>> Moving this into the PartitionNexus might not be a good idea but no
>>> problem
>>> for now we can move it later. Let me explain why:
>>> Eventually we're going to enable a root Partition with Partition nesting
>>> and
>>> so when this happens the PartitionNexus will just be another nestable
>>> Partition since these will have to handle routing based on DN to other
>>> partitions residing/nested under it.
>>> I see two possible locations for this functionality:
>>> (1) Let the InterceptorChain itself handle this since it can demarcate
>>> the
>>> start and end of calls into the chain with Txn begin and abort/commit
>>> calls.
>>> It does this by making calls against the TxnManager which I guess is the
>>> entire facade for the transaction subsystem.
>>> ---OR---
>>> (2) Handle Txn demarcation within the CoreSession. However this might not
>>> be
>>> optimal due to the need to handle additional logic which might be
>>> required
>>> for handling chain re-entry concerns.
>>> NOTE: I've not actually looked at the code after these major moves so my
>>> advice might not be very dependable. I will try to setup my environment
>>> to
>>> get a better idea of these matters.
>>> However for the time being do whatever actually makes this thing work.
>>> Let's
>>> follow an agile methodology. This thing is big. So let's get it working
>>> with
>>> solid test coverage then we can actually look at shuffling things around
>>> to
>>> optimal positions. Not saying what you've chosen is wrong ... it might
>>> just
>>> present the need for some additional refactoring when other features
>>> might
>>> need to be introduced.
>>> These are some of the biggest changes to the architecture to have taken
>>> place in years and you're doing a great job.
>>
>> You are right for txn demarcation. I wrote it wrong in the email.
>> Demarcation has to be done either at defaultcoresession or
>> defaultoperationmanager.
>
> OperationManager is probably the right place.
>
> For inner operations, I'm almost 100% sure they aren't modifiying anything
> (the only modification done as a inner operation has been removed a while
> ago : it was dealing with the addition of the ModifyTimeStamp and
> ModifiersName attribute, and it' snow done in the main operation). So we are
> safe if they are done in the current external transaction.
>
>>
>> as for where to handle the change logic, as you mentioned we need a
>> place where all interceptor chain routes end up for modification and
>> ideally we should handle modification logic above partitions using
>> master table and index interfaces so that we have a common place to
>> prepare and apply txn log edits. PartitionNexus seems to fit this
>> requirement for now. If we add another layer above PartitionNexus
>> which can get master and index table from below layers and work with
>> them, we should be able to move the change logic up there.
>>
>>>> *move xdbm-search to core as well. Making search transactional will
>>>> mostly be mostly mechanica after this point I think(hope). It should
>>>> just use the wrappers the txn manager provides for index, master and
>>>> cursors it gets from the partitions.
>>>>
>>>> *handle caches various interceptors keep. I am thinking of handling
>>>> this with a common read-write lock.
>>>>
>>> This was the latest issue for which I see some more threads. Will look at
>>> that as well.
>>
>> This is the issue I talked you about briefly. These are mostly admin
>> caches that change infrequently and that are read mostly. The
>> difficulty with them is that they are not always entry caches. They
>> might map Dns to some logical property of the entry. A simple and good
>> example is notaliascache (whether we really need this is open to
>> debate but we have it now).
>
> This is a very interesting point. The notAliasCache is definitively
> questionable. The question though is to find a better way to deal with it.
> An option would be to not support alias (yes, I know, pretty drastic, but if
> it allows us to speed up the delivery of a production ready server, I think
> this would be a good tradeoff. Also note that once we have a solid base, we
> can reintroduce alias handling in a future version).
>
>> This maps a Dn to whether the entry is an
>> alias. If we followed the normal way of merging what is read from
>> partitions with the txn log, we would have to a way of merging what is
>> read from this notaliascache with the log. This is not very difficult
>> but as we have quite a number of these caches, having a separate merge
>> and update logic for each of them is a pain and error prone.
>>
>> Instead, what I thought was to use a single system wide read-write
>> lock for these caches. Since they are read mostly, a txn might update
>> its lock to exclusive when it needs to change one of these caches and
>> might release it when it actually commits/aborts.
>
> We may start a new thread to discuss further those cache issues. We have
> many caches, for may different thinsg, and many different approaches to deal
> with the (MRU, EhCache, etc...). It's probably a good timing for clarifying
> this item...
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi !
comments inline...
On 11/1/11 12:19 PM, Selcuk AYA wrote:
> thanks for the feedback! Please see inline:
<snip/>
>
>
> I guess the xdbm-partition Maven module is now going to be pretty damn thin
> or non-existent. Was this module destroyed and if not what actually remains?
> I think some util classes for tree based implementations remain there.
> Also there some avl related classes.
We can review this project and eventually move the remaining classes
elsewhere.
>
>>> The todo is the following:
>>> *add changes to keep track of dn space changes.
>>> *test txn manager services
>>> *move the modification code path in AbstractBTreePartition in xdbm to
>>> high up to core. Probably modifications should be done in
>>> partitionNexus and partition nexus should handle preparing txn log
>>> edits and wal them. The overall flow for modifications will be:
>>> -DefaultDirectoryService:
>>> -begintxn
>>> -execute interceptor chain
>>> - handle txn abort, or conflict.
>>>
>> Moving this into the PartitionNexus might not be a good idea but no problem
>> for now we can move it later. Let me explain why:
>> Eventually we're going to enable a root Partition with Partition nesting and
>> so when this happens the PartitionNexus will just be another nestable
>> Partition since these will have to handle routing based on DN to other
>> partitions residing/nested under it.
>> I see two possible locations for this functionality:
>> (1) Let the InterceptorChain itself handle this since it can demarcate the
>> start and end of calls into the chain with Txn begin and abort/commit calls.
>> It does this by making calls against the TxnManager which I guess is the
>> entire facade for the transaction subsystem.
>> ---OR---
>> (2) Handle Txn demarcation within the CoreSession. However this might not be
>> optimal due to the need to handle additional logic which might be required
>> for handling chain re-entry concerns.
>> NOTE: I've not actually looked at the code after these major moves so my
>> advice might not be very dependable. I will try to setup my environment to
>> get a better idea of these matters.
>> However for the time being do whatever actually makes this thing work. Let's
>> follow an agile methodology. This thing is big. So let's get it working with
>> solid test coverage then we can actually look at shuffling things around to
>> optimal positions. Not saying what you've chosen is wrong ... it might just
>> present the need for some additional refactoring when other features might
>> need to be introduced.
>> These are some of the biggest changes to the architecture to have taken
>> place in years and you're doing a great job.
> You are right for txn demarcation. I wrote it wrong in the email.
> Demarcation has to be done either at defaultcoresession or
> defaultoperationmanager.
OperationManager is probably the right place.
For inner operations, I'm almost 100% sure they aren't modifiying
anything (the only modification done as a inner operation has been
removed a while ago : it was dealing with the addition of the
ModifyTimeStamp and ModifiersName attribute, and it' snow done in the
main operation). So we are safe if they are done in the current external
transaction.
>
> as for where to handle the change logic, as you mentioned we need a
> place where all interceptor chain routes end up for modification and
> ideally we should handle modification logic above partitions using
> master table and index interfaces so that we have a common place to
> prepare and apply txn log edits. PartitionNexus seems to fit this
> requirement for now. If we add another layer above PartitionNexus
> which can get master and index table from below layers and work with
> them, we should be able to move the change logic up there.
>
>>> *move xdbm-search to core as well. Making search transactional will
>>> mostly be mostly mechanica after this point I think(hope). It should
>>> just use the wrappers the txn manager provides for index, master and
>>> cursors it gets from the partitions.
>>>
>>> *handle caches various interceptors keep. I am thinking of handling
>>> this with a common read-write lock.
>>>
>> This was the latest issue for which I see some more threads. Will look at
>> that as well.
> This is the issue I talked you about briefly. These are mostly admin
> caches that change infrequently and that are read mostly. The
> difficulty with them is that they are not always entry caches. They
> might map Dns to some logical property of the entry. A simple and good
> example is notaliascache (whether we really need this is open to
> debate but we have it now).
This is a very interesting point. The notAliasCache is definitively
questionable. The question though is to find a better way to deal with
it. An option would be to not support alias (yes, I know, pretty
drastic, but if it allows us to speed up the delivery of a production
ready server, I think this would be a good tradeoff. Also note that once
we have a solid base, we can reintroduce alias handling in a future
version).
> This maps a Dn to whether the entry is an
> alias. If we followed the normal way of merging what is read from
> partitions with the txn log, we would have to a way of merging what is
> read from this notaliascache with the log. This is not very difficult
> but as we have quite a number of these caches, having a separate merge
> and update logic for each of them is a pain and error prone.
>
> Instead, what I thought was to use a single system wide read-write
> lock for these caches. Since they are read mostly, a txn might update
> its lock to exclusive when it needs to change one of these caches and
> might release it when it actually commits/aborts.
We may start a new thread to discuss further those cache issues. We have
many caches, for may different thinsg, and many different approaches to
deal with the (MRU, EhCache, etc...). It's probably a good timing for
clarifying this item...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
thanks for the feedback! Please see inline:
On Tue, Nov 1, 2011 at 5:35 AM, Alex Karasulu <ak...@apache.org> wrote:
> On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
>
> Excuse the late response. Better late than never right ...
> SNIP ...
>
>>
>> *Master table, index and index cursor wrappers: These provide
>> transactionally consistent views of entries and index data. They merge
>> what they read from their wrapped objects with the txn log manager. To
>> make this easier, txn manager keeps track of index changes for a
>> read/write txn in a summary form. One thing to note is that not all
>> information that wrappers expose is transactionall consistent. One
>> example is count methods. No need to %100 accurate on this. For these
>> cases, the wrapper just calls the wrapped object.
>>
>
> Makes perfect sense to not bother with the count methods: slight variations
> will not at all impact the optimizer's results.
>
>>
>> *For all the above, interfaces are defined in core-api and
>> implementations and internal interfaces are in core. Also moved,
>> index, indexcursor, table and related interfaces and helper classes to
>> core-api from xdbm-partition.
>>
>
> I guess the xdbm-partition Maven module is now going to be pretty damn thin
> or non-existent. Was this module destroyed and if not what actually remains?
I think some util classes for tree based implementations remain there.
Also there some avl related classes.
>
>>
>> The todo is the following:
>> *add changes to keep track of dn space changes.
>> *test txn manager services
>> *move the modification code path in AbstractBTreePartition in xdbm to
>> high up to core. Probably modifications should be done in
>> partitionNexus and partition nexus should handle preparing txn log
>> edits and wal them. The overall flow for modifications will be:
>> -DefaultDirectoryService:
>> -begintxn
>> -execute interceptor chain
>> - handle txn abort, or conflict.
>>
>
> Moving this into the PartitionNexus might not be a good idea but no problem
> for now we can move it later. Let me explain why:
> Eventually we're going to enable a root Partition with Partition nesting and
> so when this happens the PartitionNexus will just be another nestable
> Partition since these will have to handle routing based on DN to other
> partitions residing/nested under it.
> I see two possible locations for this functionality:
> (1) Let the InterceptorChain itself handle this since it can demarcate the
> start and end of calls into the chain with Txn begin and abort/commit calls.
> It does this by making calls against the TxnManager which I guess is the
> entire facade for the transaction subsystem.
> ---OR---
> (2) Handle Txn demarcation within the CoreSession. However this might not be
> optimal due to the need to handle additional logic which might be required
> for handling chain re-entry concerns.
> NOTE: I've not actually looked at the code after these major moves so my
> advice might not be very dependable. I will try to setup my environment to
> get a better idea of these matters.
> However for the time being do whatever actually makes this thing work. Let's
> follow an agile methodology. This thing is big. So let's get it working with
> solid test coverage then we can actually look at shuffling things around to
> optimal positions. Not saying what you've chosen is wrong ... it might just
> present the need for some additional refactoring when other features might
> need to be introduced.
> These are some of the biggest changes to the architecture to have taken
> place in years and you're doing a great job.
You are right for txn demarcation. I wrote it wrong in the email.
Demarcation has to be done either at defaultcoresession or
defaultoperationmanager.
as for where to handle the change logic, as you mentioned we need a
place where all interceptor chain routes end up for modification and
ideally we should handle modification logic above partitions using
master table and index interfaces so that we have a common place to
prepare and apply txn log edits. PartitionNexus seems to fit this
requirement for now. If we add another layer above PartitionNexus
which can get master and index table from below layers and work with
them, we should be able to move the change logic up there.
>>
>> *move xdbm-search to core as well. Making search transactional will
>> mostly be mostly mechanica after this point I think(hope). It should
>> just use the wrappers the txn manager provides for index, master and
>> cursors it gets from the partitions.
>>
>> *handle caches various interceptors keep. I am thinking of handling
>> this with a common read-write lock.
>>
>
> This was the latest issue for which I see some more threads. Will look at
> that as well.
This is the issue I talked you about briefly. These are mostly admin
caches that change infrequently and that are read mostly. The
difficulty with them is that they are not always entry caches. They
might map Dns to some logical property of the entry. A simple and good
example is notaliascache (whether we really need this is open to
debate but we have it now). This maps a Dn to whether the entry is an
alias. If we followed the normal way of merging what is read from
partitions with the txn log, we would have to a way of merging what is
read from this notaliascache with the log. This is not very difficult
but as we have quite a number of these caches, having a separate merge
and update logic for each of them is a pain and error prone.
Instead, what I thought was to use a single system wide read-write
lock for these caches. Since they are read mostly, a txn might update
its lock to exclusive when it needs to change one of these caches and
might release it when it actually commits/aborts.
> Again sorry about late responses.
> Regards,
> Alex
>
>>
>> regards
>> Selcuk
>>
>> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA <ay...@gmail.com> wrote:
>> > Hi all,
>> >
>> > Emmanuel wants to update the txns branch and I think it is a good time
>> > to give a good status update. Following is what I have been doing and
>> > the plan:
>> >
>> > *Log Manager: A general log manager that can be used by txns as well.
>> > Note that these logs are on disk. This is tested.
>> >
>> > *Txn Manager:Most of the services that should be provided by the txn
>> > manager is coded:
>> > begin, commit, abort, keeping
>> >
>
>
>
> --
> Best Regards,
> -- Alex
>
Re: TXN work update
Posted by Alex Karasulu <ak...@apache.org>.
On Thu, Oct 20, 2011 at 4:24 PM, Selcuk AYA <ay...@gmail.com> wrote:
Excuse the late response. Better late than never right ...
SNIP ...
> *Master table, index and index cursor wrappers: These provide
> transactionally consistent views of entries and index data. They merge
> what they read from their wrapped objects with the txn log manager. To
> make this easier, txn manager keeps track of index changes for a
> read/write txn in a summary form. One thing to note is that not all
> information that wrappers expose is transactionall consistent. One
> example is count methods. No need to %100 accurate on this. For these
> cases, the wrapper just calls the wrapped object.
>
>
Makes perfect sense to not bother with the count methods: slight variations
will not at all impact the optimizer's results.
> *For all the above, interfaces are defined in core-api and
> implementations and internal interfaces are in core. Also moved,
> index, indexcursor, table and related interfaces and helper classes to
> core-api from xdbm-partition.
>
>
I guess the xdbm-partition Maven module is now going to be pretty damn thin
or non-existent. Was this module destroyed and if not what actually remains?
> The todo is the following:
> *add changes to keep track of dn space changes.
> *test txn manager services
> *move the modification code path in AbstractBTreePartition in xdbm to
> high up to core. Probably modifications should be done in
> partitionNexus and partition nexus should handle preparing txn log
> edits and wal them. The overall flow for modifications will be:
> -DefaultDirectoryService:
> -begintxn
> -execute interceptor chain
> - handle txn abort, or conflict.
>
>
Moving this into the PartitionNexus might not be a good idea but no problem
for now we can move it later. Let me explain why:
Eventually we're going to enable a root Partition with Partition nesting
and so when this happens the PartitionNexus will just be another nestable
Partition since these will have to handle routing based on DN to other
partitions residing/nested under it.
I see two possible locations for this functionality:
(1) Let the InterceptorChain itself handle this since it can demarcate the
start and end of calls into the chain with Txn begin and abort/commit
calls. It does this by making calls against the TxnManager which I guess is
the entire facade for the transaction subsystem.
---OR---
(2) Handle Txn demarcation within the CoreSession. However this might not
be optimal due to the need to handle additional logic which might be
required for handling chain re-entry concerns.
NOTE: I've not actually looked at the code after these major moves so my
advice might not be very dependable. I will try to setup my environment to
get a better idea of these matters.
However for the time being do whatever actually makes this thing work.
Let's follow an agile methodology. This thing is big. So let's get it
working with solid test coverage then we can actually look at shuffling
things around to optimal positions. Not saying what you've chosen is wrong
... it might just present the need for some additional refactoring when
other features might need to be introduced.
These are some of the biggest changes to the architecture to have taken
place in years and you're doing a great job.
*move xdbm-search to core as well. Making search transactional will
> mostly be mostly mechanica after this point I think(hope). It should
> just use the wrappers the txn manager provides for index, master and
> cursors it gets from the partitions.
>
> *handle caches various interceptors keep. I am thinking of handling
> this with a common read-write lock.
>
>
This was the latest issue for which I see some more threads. Will look at
that as well.
Again sorry about late responses.
Regards,
Alex
> regards
> Selcuk
>
> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA <ay...@gmail.com> wrote:
> > Hi all,
> >
> > Emmanuel wants to update the txns branch and I think it is a good time
> > to give a good status update. Following is what I have been doing and
> > the plan:
> >
> > *Log Manager: A general log manager that can be used by txns as well.
> > Note that these logs are on disk. This is tested.
> >
> > *Txn Manager:Most of the services that should be provided by the txn
> > manager is coded:
> > begin, commit, abort, keeping
> >
>
--
Best Regards,
-- Alex
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
Thanks a lot for the heads up, Selcuk !
I'm done with the merge. It was painful, as expected. Many new projects
has been added, and some packages have been moved (the .api. prefix has
been added for partition and many other classes.
I hope I haven't broke too much things... I tried to build the
apacheds-txns, and git a success, so I'm quite confident :)
On 10/20/11 3:24 PM, Selcuk AYA wrote:
> Continuing the previous incomplete email:)
> Emmanuel wants to update the txns branch and I think it is a good time
> to give a good status update. Following is what I have been doing and
> the plan:
>
> *Log Manager: A general log manager that can be used by txns as well.
> Note that these logs are on disk. This is tested.
>
> *Txn and Txn log manager:Most of the services that should be provided
> by the txn manager is coded:
> *begin, commit, abort of txns
> * keeping track of changes for read-write txns.
> *expose interface for txns to wal their changes.
> One service missing here is keeping track of the read/changed Dn set
> to do txn verification and conflict detection for read-write txn.
>
> *Master table, index and index cursor wrappers: These provide
> transactionally consistent views of entries and index data. They merge
> what they read from their wrapped objects with the txn log manager. To
> make this easier, txn manager keeps track of index changes for a
> read/write txn in a summary form. One thing to note is that not all
> information that wrappers expose is transactionall consistent. One
> example is count methods. No need to %100 accurate on this. For these
> cases, the wrapper just calls the wrapped object.
>
> *For all the above, interfaces are defined in core-api and
> implementations and internal interfaces are in core. Also moved,
> index, indexcursor, table and related interfaces and helper classes to
> core-api from xdbm-partition.
>
> The todo is the following:
> *add changes to keep track of dn space changes.
> *test txn manager services
> *move the modification code path in AbstractBTreePartition in xdbm to
> high up to core. Probably modifications should be done in
> partitionNexus and partition nexus should handle preparing txn log
> edits and wal them. The overall flow for modifications will be:
> -DefaultDirectoryService:
> -begintxn
> -execute interceptor chain
> - handle txn abort, or conflict.
>
> *move xdbm-search to core as well. Making search transactional will
> mostly be mostly mechanica after this point I think(hope). It should
> just use the wrappers the txn manager provides for index, master and
> cursors it gets from the partitions.
>
> *handle caches various interceptors keep. I am thinking of handling
> this with a common read-write lock.
>
> regards
> Selcuk
>
> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA<ay...@gmail.com> wrote:
>> Hi all,
>>
>> Emmanuel wants to update the txns branch and I think it is a good time
>> to give a good status update. Following is what I have been doing and
>> the plan:
>>
>> *Log Manager: A general log manager that can be used by txns as well.
>> Note that these logs are on disk. This is tested.
>>
>> *Txn Manager:Most of the services that should be provided by the txn
>> manager is coded:
>> begin, commit, abort, keeping
>>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 10/20/11 3:24 PM, Selcuk AYA wrote:
> Continuing the previous incomplete email:)
> Emmanuel wants to update the txns branch and I think it is a good time
> to give a good status update. Following is what I have been doing and
> the plan:
>
> *Log Manager: A general log manager that can be used by txns as well.
> Note that these logs are on disk. This is tested.
>
> *Txn and Txn log manager:Most of the services that should be provided
> by the txn manager is coded:
> *begin, commit, abort of txns
> * keeping track of changes for read-write txns.
> *expose interface for txns to wal their changes.
> One service missing here is keeping track of the read/changed Dn set
> to do txn verification and conflict detection for read-write txn.
>
> *Master table, index and index cursor wrappers: These provide
> transactionally consistent views of entries and index data. They merge
> what they read from their wrapped objects with the txn log manager. To
> make this easier, txn manager keeps track of index changes for a
> read/write txn in a summary form. One thing to note is that not all
> information that wrappers expose is transactionall consistent. One
> example is count methods. No need to %100 accurate on this. For these
> cases, the wrapper just calls the wrapped object.
>
> *For all the above, interfaces are defined in core-api and
> implementations and internal interfaces are in core. Also moved,
> index, indexcursor, table and related interfaces and helper classes to
> core-api from xdbm-partition.
>
> The todo is the following:
> *add changes to keep track of dn space changes.
> *test txn manager services
> *move the modification code path in AbstractBTreePartition in xdbm to
> high up to core. Probably modifications should be done in
> partitionNexus and partition nexus should handle preparing txn log
> edits and wal them. The overall flow for modifications will be:
> -DefaultDirectoryService:
> -begintxn
> -execute interceptor chain
> - handle txn abort, or conflict.
>
> *move xdbm-search to core as well. Making search transactional will
> mostly be mostly mechanica after this point I think(hope). It should
> just use the wrappers the txn manager provides for index, master and
> cursors it gets from the partitions.
>
> *handle caches various interceptors keep. I am thinking of handling
> this with a common read-write lock.
>
> regards
> Selcuk
>
> On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA<ay...@gmail.com> wrote:
>> Hi all,
>>
>> Emmanuel wants to update the txns branch and I think it is a good time
>> to give a good status update. Following is what I have been doing and
>> the plan:
>>
>> *Log Manager: A general log manager that can be used by txns as well.
>> Note that these logs are on disk. This is tested.
>>
>> *Txn Manager:Most of the services that should be provided by the txn
>> manager is coded:
>> begin, commit, abort, keeping
>>
Awsome !
I'm going to merge the changes we have done in trunk into your branch.
It should not have too much impacts, as you are working in some
different areas.
At least, it will alleviate us the pain to do it when the branch will be
ready.
Many thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: TXN work update
Posted by Selcuk AYA <ay...@gmail.com>.
Continuing the previous incomplete email:)
Emmanuel wants to update the txns branch and I think it is a good time
to give a good status update. Following is what I have been doing and
the plan:
*Log Manager: A general log manager that can be used by txns as well.
Note that these logs are on disk. This is tested.
*Txn and Txn log manager:Most of the services that should be provided
by the txn manager is coded:
*begin, commit, abort of txns
* keeping track of changes for read-write txns.
*expose interface for txns to wal their changes.
One service missing here is keeping track of the read/changed Dn set
to do txn verification and conflict detection for read-write txn.
*Master table, index and index cursor wrappers: These provide
transactionally consistent views of entries and index data. They merge
what they read from their wrapped objects with the txn log manager. To
make this easier, txn manager keeps track of index changes for a
read/write txn in a summary form. One thing to note is that not all
information that wrappers expose is transactionall consistent. One
example is count methods. No need to %100 accurate on this. For these
cases, the wrapper just calls the wrapped object.
*For all the above, interfaces are defined in core-api and
implementations and internal interfaces are in core. Also moved,
index, indexcursor, table and related interfaces and helper classes to
core-api from xdbm-partition.
The todo is the following:
*add changes to keep track of dn space changes.
*test txn manager services
*move the modification code path in AbstractBTreePartition in xdbm to
high up to core. Probably modifications should be done in
partitionNexus and partition nexus should handle preparing txn log
edits and wal them. The overall flow for modifications will be:
-DefaultDirectoryService:
-begintxn
-execute interceptor chain
- handle txn abort, or conflict.
*move xdbm-search to core as well. Making search transactional will
mostly be mostly mechanica after this point I think(hope). It should
just use the wrappers the txn manager provides for index, master and
cursors it gets from the partitions.
*handle caches various interceptors keep. I am thinking of handling
this with a common read-write lock.
regards
Selcuk
On Thu, Oct 20, 2011 at 4:05 PM, Selcuk AYA <ay...@gmail.com> wrote:
> Hi all,
>
> Emmanuel wants to update the txns branch and I think it is a good time
> to give a good status update. Following is what I have been doing and
> the plan:
>
> *Log Manager: A general log manager that can be used by txns as well.
> Note that these logs are on disk. This is tested.
>
> *Txn Manager:Most of the services that should be provided by the txn
> manager is coded:
> begin, commit, abort, keeping
>