You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lecharny <el...@gmail.com> on 2011/01/15 12:01:22 UTC
Some more thoughts about AP handling
Hi guys,
I'd like to share some thoughts I have had yesterday and this morning
while I was lying in my bed (one of the best place to find solutions to
problems with the shower).
AP support suppose you deal with some constraints, as explicited in
various RFCs :
- we must support SAP and IAP, otherwise it's too complicated to manage
ACIs when they are overlapping rights
- Subentries must be referenced in selected entries as a DN
- we must cover at least 3 administrative role : ACI,
Collective-Attributes, Subschema, two of those aspects being described
in RFCs
- every operation must be atomic, ie either execute *fully* or be
aborted *fully*. We can't have a partial execution.
In order to correctly implement APs, we don't have many possible
solutions. Let me list them again :
#0 don't support AP at all. Well, it's not really an option :)
#1 evaluate entries on the fly when we access them, all the time
#2 evaluate entries on the fly the first time we access them, and store
the result
#3 evaluate entries when the AP is modified, and persist the state
All the 3 last approaches have pros and cons. I have ruled out option
#0, so let's analyze the three others
#1 pros :
- atomicity is easy to enforce : we always do one single modification
when managing APs
- it's easy to implement, as we just have to handle subentries in the
search and lookup methods
- we don't have any operation that could impact the whole set of entries
#1 cons :
- we will have to pay the price of the entry evaluation for every single
read we do on any entry
#2 pros :
- atomicity is somehow easier to enforce, but we still have some case
where we have to update more than one entry when managing APs
- potentially let the system in a consistent state if it crashes
#2 cons :
- very complex to implement
- there are some cases (like for the rename/move operations) where we
either decide to impact a set of entries for nothing, or we have to
immediately update a set of entries, defeating one of the [pros]
- the first reader pay the price
#3 pros :
- the time it costs to update the entries is paid only once
- update only the set of impacted entries
#3 cons :
- atomicity is not enforced, if the server crashes in the middle of an
operation, it's unstable
- it can take forever to update a huge base, assuming that we have to
update all the entries, and it make take a lot of time at 1000 entries
written per second
- implementation is complex, but less than for #2
At this point, I'm facing some big issue... If I want the rename/move
operations implemented for option #2, it will imply either that we might
impact many more entries than necessary, as I have to tell the parent AP
that it's not up to date, *or* I have to adopt the #3 solution and
update immediately the children entries...
Ok, let's face the facts : maybe option #1 is the only possible solution
we have atm, from different aspects :
- delay : implementing option #1 is probably a matter of one week
- atomicity : it's the only solution that guarantees it
The price to pay is a slower server for search operations, as each entry
will be evaluated again and again.
I don't know about you, but (running away from such things like "I feel
in my guts", "I believe that", etc) from the technical point of view and
the schedule point of view, I think that what matters right now are the
following points :
- we need to get the AP working fully, following the specifications and
constraints
- we need to get a 2.0-M1 out to be able to move to the next step
- we need to implement a Transaction Manager in 2.0-M1 in order to
implement a better strategy (ie #2 or #3)
Really the problem now, and this is what I was trying to workaround with
option #2, is the atomicity constraint. We need a transaction manager
badly, and we need it for 2.0, no matter what, before we can implement
either #2 or #3. And once we get this feature added in the server, then
I do think that #3 is the best possible option.
I'll discuss the TM (Transaction Manager) in another thread, I just want
to know what's your opinion about this aspect.
so, wdyt ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Emmanuel Lécharny <el...@apache.org>.
On 1/16/11 2:51 PM, Alex Karasulu wrote:
>> Then we will be able to start discussing about what is in M2, then in M3,
>> etc... Let's not anticipate on the problem we will face anyway before facing
>> them :)
> That's up to you and this certainly is a less expensive way to
> approach it. Everyone has their own way to approach things. My input
> was optional, please disregard it.
Note that I didn't want to stop any discussion, or any proposal. I just
wanted to express that I'm really exited about having M1 out, and that
once it will be done, we can move on M2 and see what we put inside. I
must admit I'm more focused on short term targets atm, because it's
months we have released 1.5.7, and we badly need a new release asap.
PS : In fact, I need a short break, and I want to get it after 2.0-M1.
One week off would be really a good thing for me, I'm exhausted...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Alex Karasulu <ak...@apache.org>.
On Sun, Jan 16, 2011 at 3:44 PM, Emmanuel Lécharny <el...@apache.org> wrote:
> On 1/16/11 1:14 PM, Alex Karasulu wrote:
>>
>>> Now, this is true for updates, not for searches which span over more than
>>> one entry, which means the entries you get back may not be fresh when you
>>> want to use them.
>>
>> Isolation is non-existant allowing for dirty reads. We can solve this
>> issue though. It's going to take time and some evolution.
>>
>>> However, ADS has a few issues with ACIDity when it comes to manage APs,
>>> as
>>> we update more than one entry when injecting a subentry. (I don't think
>>> there are other oparations that need more than one update, but OTOH, a
>>> lot
>>> of update operations do a lookup before doing a write).
>>
>> SPs and Triggers will present these same issues.
>
> Very true. This is an area we have to investigate, as it's more a prototype
> right now than a completed feature.
+1 ... Absolutely!
>>
>> AP is the first time we seriously hit a wall with this because we're
>> getting to complex scenarios that are requiring these multiple updates
>> to happen in an atomic way. There will be more cases in the future I'm
>> almost certain of this.
>
> FTR, we fixed some ACID issues last year, when we reviewed all the
> operations. The operational attributes were added after the entry was stored
> in the
> backend, leading to a potential breakage. Now, the op attrs are injecting in
> the entry *before* being injected in the backend.
Yeah that was a good improvisation to handle the lack of transaction atomicity.
> This was an issue we detected 4 years ago :
> https://issues.apache.org/jira/browse/DIRSERVER-840
IMHO we were very lucky that we had this opportunity. We're being as
creative as possible with the lack of transaction support. We just
need to bite the bullet and get this feature done once and for all.
>>> This is the reason we need a TM. Note that if we don't update the entries
>>> when injecting a subentry, we are safe (this is option #1, btw)
>>>>
>>>> Then a set of M2 releases can be made.
>>>
>>> I would move that for M3 instead. We don't have a lot of background on TM
>>> atm.
>>
>> Actually this sounds a lot more difficult than it will be. We also
>> have a lot of the machinery available. We need txn log and some other
>> things.
>>
>> However what I recommend is to do this in steps. Does not have to be
>> fully implemented in a single milestone release. We can just erect the
>> scaffolding and have the rest of the system using the API.
>>
>> Across other milestones we can start filling in the implementation
>> with minimal impact to ADS SPI users.
>
> <Snip/>
>
> Right now, before discussing what will be in M7, I think the best is to get
> M1 out with OSGi, some refactoring in shared, and as few possible thing as
> possible !
+1
> Then we will be able to start discussing about what is in M2, then in M3,
> etc... Let's not anticipate on the problem we will face anyway before facing
> them :)
That's up to you and this certainly is a less expensive way to
approach it. Everyone has their own way to approach things. My input
was optional, please disregard it.
Regards,
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu
Re: Some more thoughts about AP handling
Posted by Emmanuel Lécharny <el...@apache.org>.
On 1/16/11 1:14 PM, Alex Karasulu wrote:
>
>> Now, this is true for updates, not for searches which span over more than
>> one entry, which means the entries you get back may not be fresh when you
>> want to use them.
> Isolation is non-existant allowing for dirty reads. We can solve this
> issue though. It's going to take time and some evolution.
>
>> However, ADS has a few issues with ACIDity when it comes to manage APs, as
>> we update more than one entry when injecting a subentry. (I don't think
>> there are other oparations that need more than one update, but OTOH, a lot
>> of update operations do a lookup before doing a write).
> SPs and Triggers will present these same issues.
Very true. This is an area we have to investigate, as it's more a
prototype right now than a completed feature.
> AP is the first time we seriously hit a wall with this because we're
> getting to complex scenarios that are requiring these multiple updates
> to happen in an atomic way. There will be more cases in the future I'm
> almost certain of this.
FTR, we fixed some ACID issues last year, when we reviewed all the
operations. The operational attributes were added after the entry was
stored in the
backend, leading to a potential breakage. Now, the op attrs are
injecting in the entry *before* being injected in the backend.
This was an issue we detected 4 years ago :
https://issues.apache.org/jira/browse/DIRSERVER-840
>> This is the reason we need a TM. Note that if we don't update the entries
>> when injecting a subentry, we are safe (this is option #1, btw)
>>> Then a set of M2 releases can be made.
>> I would move that for M3 instead. We don't have a lot of background on TM
>> atm.
> Actually this sounds a lot more difficult than it will be. We also
> have a lot of the machinery available. We need txn log and some other
> things.
>
> However what I recommend is to do this in steps. Does not have to be
> fully implemented in a single milestone release. We can just erect the
> scaffolding and have the rest of the system using the API.
>
> Across other milestones we can start filling in the implementation
> with minimal impact to ADS SPI users.
<Snip/>
Right now, before discussing what will be in M7, I think the best is to
get M1 out with OSGi, some refactoring in shared, and as few possible
thing as possible !
Then we will be able to start discussing about what is in M2, then in
M3, etc... Let's not anticipate on the problem we will face anyway
before facing them :)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Alex Karasulu <ak...@apache.org>.
On Sun, Jan 16, 2011 at 1:23 PM, Emmanuel Lécharny <el...@apache.org> wrote:
> On 1/16/11 11:52 AM, Alex Karasulu wrote:
...
>>> - implement #1 : no more modifications made on the backend, all is
>>> evaluated
>>> on the fly
>>
>> After this I suggest we clean up shared, and get Studio depending
>> directly on it's OSGi bundles, and do some M1 releases. This can be
>> the first 2-3 week sprint.
>
> +1. But I think it can be even faster.
Wonderful if we can do that, even better.
>>> - have a TransactionManager (TM) implemented
>>
>> Before this point we can cleanup some of the ADS SPIs and then
>> refactor the interceptor chain making transaction management a core
>> function of the chain rather than an interceptor. There should be no
>> degrees of freedom with this feature since it's a must have.
>
> This is a complex situation. We all agree that it's a core function, and
> should always be present and at the end of the chain if it remains an
> interceptor. But this is a larger problem : we have other interceptors in
> the chain that must be at specific positions and should always be present.
Yes this is what is drawing me closer to the option of integrating
this feature directly into the chain. The interactions here are
getting complicated, and we may need to get a bit explicit about this
capability by directly building it into the chain.
This way there are no worries about where in the chain anything goes.
I don't want another interceptor even thinking about messing with how
this mechanism works. It should not be something that is left open to
tampering by even an Interceptor implementor.
We can create interfaces where if users want to replace the TxNManager
or it's component they can do so. But the relationship it has in the
chain is an explicit contract not left to any interpretation or
ambiguity.
> We can discuss those points after M1.
Sure. I like to think a couple steps ahead in case we can inject into
M1 things that help rather than make these features put into M2 more
difficult.
I agree 100% with the one step at a time approach: dropping some hints
so people can chew on these thoughts over time.
>> TxN management although not imparting ACID capabilities in this round,
>> should be clear, clean, and simple with all scaffolding erected so the
>> API can be properly used. Slowly we can then add all the
>> implementation details needed to implement all ACID features with
>> coming milestones.
>
> Right now, LDAP servers are just supposed to guarantee atomicity,
> Consistency and Durability. As all the operations are supposed to affect
> only *one* entry, this leads LDAP implementation to implement Isolation as
> well (so it's full ACID).
No one would disagree with these points. We do need all ACID features
and I feel we're not that far from reaching them.
> Now, this is true for updates, not for searches which span over more than
> one entry, which means the entries you get back may not be fresh when you
> want to use them.
Isolation is non-existant allowing for dirty reads. We can solve this
issue though. It's going to take time and some evolution.
> However, ADS has a few issues with ACIDity when it comes to manage APs, as
> we update more than one entry when injecting a subentry. (I don't think
> there are other oparations that need more than one update, but OTOH, a lot
> of update operations do a lookup before doing a write).
SPs and Triggers will present these same issues.
AP is the first time we seriously hit a wall with this because we're
getting to complex scenarios that are requiring these multiple updates
to happen in an atomic way. There will be more cases in the future I'm
almost certain of this.
> This is the reason we need a TM. Note that if we don't update the entries
> when injecting a subentry, we are safe (this is option #1, btw)
>>
>> Then a set of M2 releases can be made.
>
> I would move that for M3 instead. We don't have a lot of background on TM
> atm.
Actually this sounds a lot more difficult than it will be. We also
have a lot of the machinery available. We need txn log and some other
things.
However what I recommend is to do this in steps. Does not have to be
fully implemented in a single milestone release. We can just erect the
scaffolding and have the rest of the system using the API.
Across other milestones we can start filling in the implementation
with minimal impact to ADS SPI users.
>>>
>>> - add support for IAPs
>>> - once all those pieces have been added, then we can move to #3
>>
>> This can be an M3 release.
>
> M3 release could contain the TM, and a M4 a complete ACID implementation of
> APs, where we update entries instead of evaluating them on the fly.
You and I both know we can set features to be in milestones and it all
shifts. I seldom works for us. Let's be a little bit more flexible but
make sure we're pumping stuff out with milestones.
It would be nice if we got into a groove every 2-3 weeks to pump out
another milestone release even if features are just partially
implemented.
If [A|S]PI's are solid we can introduce new implementation elements
with the minimum of disruption to our users.
>>>
>>> Of course, we have other things on our plate which can be done in
>>> parallel,
>>> like refactoring the APIs.
>>
>> There will probably be incremental steps here. For M1 we can do what
>> seems reasonable and clear to us. Like a base line push, but I am sure
>> we'll begin to see more refactorings condense as we progress on each
>> sprint.
>
> In fact, having a M1 fast could be a good first step, considering that it
> will simply expose the current trunk and the new configuration system.
+1 there. Nice to have users gradually getting into the groove of
using the LDAP based configuration. We get more feedback and can
incorporate that into upcoming milestone releases.
There
> are some documentation to be written in this area, and it's hard to do so on
> a moving target. Also for studio 2.0-M1, which include a plugin to manage
> ADS 2.0 config, having a stable release of ADS would be a great thing.
Absolutely, shifting APIs sucks for us (since we use it) but sucks
even more for our users. The milestone scheme is great though, since
users know that they're promised API stability with RC1. Still though
we can take certain steps to make API shifts as painless as possible,
like by getting these interfaces carved out, released and continuing
implementation enhancements in the background.
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu
Re: Some more thoughts about AP handling
Posted by Emmanuel Lécharny <el...@apache.org>.
On 1/16/11 11:52 AM, Alex Karasulu wrote:
> - first make the current trunk working. We have an issue as the subentries
>> are not read back when the server is stopped, so we can't have the ACI
>> working again. That's a stupid error which can be fixed easily.
I *think* I have fixed the issue last night (there was a bad removal in
the APCache).
I will conduct some more thorough tests this afternoon.
>> - implement #1 : no more modifications made on the backend, all is evaluated
>> on the fly
> After this I suggest we clean up shared, and get Studio depending
> directly on it's OSGi bundles, and do some M1 releases. This can be
> the first 2-3 week sprint.
+1. But I think it can be even faster.
>> - have a TransactionManager (TM) implemented
> Before this point we can cleanup some of the ADS SPIs and then
> refactor the interceptor chain making transaction management a core
> function of the chain rather than an interceptor. There should be no
> degrees of freedom with this feature since it's a must have.
This is a complex situation. We all agree that it's a core function, and
should always be present and at the end of the chain if it remains an
interceptor. But this is a larger problem : we have other interceptors
in the chain that must be at specific positions and should always be
present.
We can discuss those points after M1.
> The two journals (CL& Journal) should be consolidated into a single
> log even if two separate tactics are used inside the same component.
> This means the CL and Journal are merged and no longer managed using
> Interceptors. There are other cleanup we can do to simplify the chain
> as well.
+1 for merging them.
> TxN management although not imparting ACID capabilities in this round,
> should be clear, clean, and simple with all scaffolding erected so the
> API can be properly used. Slowly we can then add all the
> implementation details needed to implement all ACID features with
> coming milestones.
Right now, LDAP servers are just supposed to guarantee atomicity,
Consistency and Durability. As all the operations are supposed to affect
only *one* entry, this leads LDAP implementation to implement Isolation
as well (so it's full ACID).
Now, this is true for updates, not for searches which span over more
than one entry, which means the entries you get back may not be fresh
when you want to use them.
However, ADS has a few issues with ACIDity when it comes to manage APs,
as we update more than one entry when injecting a subentry. (I don't
think there are other oparations that need more than one update, but
OTOH, a lot of update operations do a lookup before doing a write).
This is the reason we need a TM. Note that if we don't update the
entries when injecting a subentry, we are safe (this is option #1, btw)
> Then a set of M2 releases can be made.
I would move that for M3 instead. We don't have a lot of background on
TM atm.
>> - add support for IAPs
>> - once all those pieces have been added, then we can move to #3
> This can be an M3 release.
M3 release could contain the TM, and a M4 a complete ACID implementation
of APs, where we update entries instead of evaluating them on the fly.
>> Of course, we have other things on our plate which can be done in parallel,
>> like refactoring the APIs.
> There will probably be incremental steps here. For M1 we can do what
> seems reasonable and clear to us. Like a base line push, but I am sure
> we'll begin to see more refactorings condense as we progress on each
> sprint.
In fact, having a M1 fast could be a good first step, considering that
it will simply expose the current trunk and the new configuration
system. There are some documentation to be written in this area, and
it's hard to do so on a moving target. Also for studio 2.0-M1, which
include a plugin to manage ADS 2.0 config, having a stable release of
ADS would be a great thing.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Alex Karasulu <ak...@apache.org>.
On Sat, Jan 15, 2011 at 3:44 PM, Emmanuel Lécharny <el...@apache.org> wrote:
> On 1/15/11 1:59 PM, Alex Karasulu wrote:
>>
>> On Sat, Jan 15, 2011 at 1:01 PM, Emmanuel Lecharny<el...@gmail.com>
>> wrote:
>>>
>>> Hi guys,
>>>
>>> I'd like to share some thoughts I have had yesterday and this morning
>>> while
>>> I was lying in my bed (one of the best place to find solutions to
>>> problems
>>> with the shower).
>>>
>>> AP support suppose you deal with some constraints, as explicited in
>>> various
>>> RFCs :
>>> - we must support SAP and IAP, otherwise it's too complicated to manage
>>> ACIs
>>> when they are overlapping rights
>>> - Subentries must be referenced in selected entries as a DN
>>> - we must cover at least 3 administrative role : ACI,
>>> Collective-Attributes,
>>> Subschema, two of those aspects being described in RFCs
>>> - every operation must be atomic, ie either execute *fully* or be aborted
>>> *fully*. We can't have a partial execution.
>>>
>>> In order to correctly implement APs, we don't have many possible
>>> solutions.
>>> Let me list them again :
>>> #0 don't support AP at all. Well, it's not really an option :)
>>> #1 evaluate entries on the fly when we access them, all the time
>>> #2 evaluate entries on the fly the first time we access them, and store
>>> the
>>> result
>>> #3 evaluate entries when the AP is modified, and persist the state
>>>
>>> All the 3 last approaches have pros and cons. I have ruled out option #0,
>>> so
>>> let's analyze the three others
>>>
>>> #1 pros :
>>> - atomicity is easy to enforce : we always do one single modification
>>> when
>>> managing APs
>>> - it's easy to implement, as we just have to handle subentries in the
>>> search
>>> and lookup methods
>>> - we don't have any operation that could impact the whole set of entries
>>
>> Also must ask, when search touches entries and updates them in this
>> lazy update mechanism, do we replicate the changes or these are
>> changes that never get replicated? I would think the later.
>
> This is #2. In #1, the entries are not modified at all on disk.
>>>
>>> #1 cons :
>>> - we will have to pay the price of the entry evaluation for every single
>>> read we do on any entry
>>
>> Every single first read correct?
>
> correct.
> <snip/>
>>>
>>> , no matter what, before we can implement either #2 or
>>> #3. And once we get this feature added in the server, then I do think
>>> that
>>> #3 is the best possible option.
>>
>> Yes yes yes ! I totally agree.
>>
>> Some things really cannot progress unless some supporting
>> infrastructure is laid down. The AP subsystem is exactly suffering
>> from this. Everything must evolve together. Otherwise we wind up
>> implementing a lot of forced solutions that are less than optimal with
>> hacks, work arounds and one offs. Things get more complex and brittle
>> causing more pain down the line.
>>
>> Let's switch into a mode of cleaning up the shop after all this heavy
>> sawing and hammering. Sit down and plan a micro strategy with a clear
>> mind and clean shop. Then knock this out in a solid and easy way.
>>
>> The AP thing was getting way too hard. I commend you on your effort,
>> and you're close but it's almost impossible to do without a local
>> transaction manager. Definitely impossible to do RIGHT without one.
>
> The main issue here is that I was trying to catch many birds with one stone.
> Not wanting to implement a transaction manager made #2 solution the only one
> available, but it's not a perfect one.
>
> Right now, I really think we should go step by step, following the following
> path (roughly) :
I recommend taking an agile approach to the items below with 2-3 week
sprints each followed by a milestone release.
> - first make the current trunk working. We have an issue as the subentries
> are not read back when the server is stopped, so we can't have the ACI
> working again. That's a stupid error which can be fixed easily.
> - implement #1 : no more modifications made on the backend, all is evaluated
> on the fly
After this I suggest we clean up shared, and get Studio depending
directly on it's OSGi bundles, and do some M1 releases. This can be
the first 2-3 week sprint.
> - have a TransactionManager (TM) implemented
Before this point we can cleanup some of the ADS SPIs and then
refactor the interceptor chain making transaction management a core
function of the chain rather than an interceptor. There should be no
degrees of freedom with this feature since it's a must have.
The two journals (CL & Journal) should be consolidated into a single
log even if two separate tactics are used inside the same component.
This means the CL and Journal are merged and no longer managed using
Interceptors. There are other cleanup we can do to simplify the chain
as well.
TxN management although not imparting ACID capabilities in this round,
should be clear, clean, and simple with all scaffolding erected so the
API can be properly used. Slowly we can then add all the
implementation details needed to implement all ACID features with
coming milestones.
Then a set of M2 releases can be made.
> - add support for IAPs
> - once all those pieces have been added, then we can move to #3
This can be an M3 release.
> Of course, we have other things on our plate which can be done in parallel,
> like refactoring the APIs.
There will probably be incremental steps here. For M1 we can do what
seems reasonable and clear to us. Like a base line push, but I am sure
we'll begin to see more refactorings condense as we progress on each
sprint.
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu
Re: Some more thoughts about AP handling
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 1/15/11 7:16 PM, Stefan Seelmann wrote:
> On Sat, Jan 15, 2011 at 2:44 PM, Emmanuel Lécharny<el...@apache.org> wrote:
>> Right now, I really think we should go step by step, following the following
>> path (roughly) :
>> - first make the current trunk working. We have an issue as the subentries
>> are not read back when the server is stopped, so we can't have the ACI
>> working again. That's a stupid error which can be fixed easily.
>> - implement #1 : no more modifications made on the backend, all is evaluated
>> on the fly
>> - have a TransactionManager (TM) implemented
>> - add support for IAPs
>> - once all those pieces have been added, then we can move to #3
> Sounds reasonable.
I have checked that the subentries are reloaded. I have a few more
checks to run.
We may want to release fast in order to be able to move to a next step.
Will update the list soon.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Stefan Seelmann <se...@apache.org>.
On Sat, Jan 15, 2011 at 2:44 PM, Emmanuel Lécharny <el...@apache.org> wrote:
> Right now, I really think we should go step by step, following the following
> path (roughly) :
> - first make the current trunk working. We have an issue as the subentries
> are not read back when the server is stopped, so we can't have the ACI
> working again. That's a stupid error which can be fixed easily.
> - implement #1 : no more modifications made on the backend, all is evaluated
> on the fly
> - have a TransactionManager (TM) implemented
> - add support for IAPs
> - once all those pieces have been added, then we can move to #3
Sounds reasonable.
Kind Regards,
Stefan
Re: Some more thoughts about AP handling
Posted by Emmanuel Lécharny <el...@apache.org>.
On 1/15/11 1:59 PM, Alex Karasulu wrote:
> On Sat, Jan 15, 2011 at 1:01 PM, Emmanuel Lecharny<el...@gmail.com> wrote:
>> Hi guys,
>>
>> I'd like to share some thoughts I have had yesterday and this morning while
>> I was lying in my bed (one of the best place to find solutions to problems
>> with the shower).
>>
>> AP support suppose you deal with some constraints, as explicited in various
>> RFCs :
>> - we must support SAP and IAP, otherwise it's too complicated to manage ACIs
>> when they are overlapping rights
>> - Subentries must be referenced in selected entries as a DN
>> - we must cover at least 3 administrative role : ACI, Collective-Attributes,
>> Subschema, two of those aspects being described in RFCs
>> - every operation must be atomic, ie either execute *fully* or be aborted
>> *fully*. We can't have a partial execution.
>>
>> In order to correctly implement APs, we don't have many possible solutions.
>> Let me list them again :
>> #0 don't support AP at all. Well, it's not really an option :)
>> #1 evaluate entries on the fly when we access them, all the time
>> #2 evaluate entries on the fly the first time we access them, and store the
>> result
>> #3 evaluate entries when the AP is modified, and persist the state
>>
>> All the 3 last approaches have pros and cons. I have ruled out option #0, so
>> let's analyze the three others
>>
>> #1 pros :
>> - atomicity is easy to enforce : we always do one single modification when
>> managing APs
>> - it's easy to implement, as we just have to handle subentries in the search
>> and lookup methods
>> - we don't have any operation that could impact the whole set of entries
> Also must ask, when search touches entries and updates them in this
> lazy update mechanism, do we replicate the changes or these are
> changes that never get replicated? I would think the later.
This is #2. In #1, the entries are not modified at all on disk.
>> #1 cons :
>> - we will have to pay the price of the entry evaluation for every single
>> read we do on any entry
> Every single first read correct?
correct.
<snip/>
>> , no matter what, before we can implement either #2 or
>> #3. And once we get this feature added in the server, then I do think that
>> #3 is the best possible option.
> Yes yes yes ! I totally agree.
>
> Some things really cannot progress unless some supporting
> infrastructure is laid down. The AP subsystem is exactly suffering
> from this. Everything must evolve together. Otherwise we wind up
> implementing a lot of forced solutions that are less than optimal with
> hacks, work arounds and one offs. Things get more complex and brittle
> causing more pain down the line.
>
> Let's switch into a mode of cleaning up the shop after all this heavy
> sawing and hammering. Sit down and plan a micro strategy with a clear
> mind and clean shop. Then knock this out in a solid and easy way.
>
> The AP thing was getting way too hard. I commend you on your effort,
> and you're close but it's almost impossible to do without a local
> transaction manager. Definitely impossible to do RIGHT without one.
The main issue here is that I was trying to catch many birds with one
stone. Not wanting to implement a transaction manager made #2 solution
the only one available, but it's not a perfect one.
Right now, I really think we should go step by step, following the
following path (roughly) :
- first make the current trunk working. We have an issue as the
subentries are not read back when the server is stopped, so we can't
have the ACI working again. That's a stupid error which can be fixed easily.
- implement #1 : no more modifications made on the backend, all is
evaluated on the fly
- have a TransactionManager (TM) implemented
- add support for IAPs
- once all those pieces have been added, then we can move to #3
Of course, we have other things on our plate which can be done in
parallel, like refactoring the APIs.
That just some options we can discuss atm. Let's use this week-end
discussing, and start back with a better roadmap later next week.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some more thoughts about AP handling
Posted by Alex Karasulu <ak...@apache.org>.
On Sat, Jan 15, 2011 at 1:01 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> Hi guys,
>
> I'd like to share some thoughts I have had yesterday and this morning while
> I was lying in my bed (one of the best place to find solutions to problems
> with the shower).
>
> AP support suppose you deal with some constraints, as explicited in various
> RFCs :
> - we must support SAP and IAP, otherwise it's too complicated to manage ACIs
> when they are overlapping rights
> - Subentries must be referenced in selected entries as a DN
> - we must cover at least 3 administrative role : ACI, Collective-Attributes,
> Subschema, two of those aspects being described in RFCs
> - every operation must be atomic, ie either execute *fully* or be aborted
> *fully*. We can't have a partial execution.
>
> In order to correctly implement APs, we don't have many possible solutions.
> Let me list them again :
> #0 don't support AP at all. Well, it's not really an option :)
> #1 evaluate entries on the fly when we access them, all the time
> #2 evaluate entries on the fly the first time we access them, and store the
> result
> #3 evaluate entries when the AP is modified, and persist the state
>
> All the 3 last approaches have pros and cons. I have ruled out option #0, so
> let's analyze the three others
>
> #1 pros :
> - atomicity is easy to enforce : we always do one single modification when
> managing APs
> - it's easy to implement, as we just have to handle subentries in the search
> and lookup methods
> - we don't have any operation that could impact the whole set of entries
Also must ask, when search touches entries and updates them in this
lazy update mechanism, do we replicate the changes or these are
changes that never get replicated? I would think the later.
> #1 cons :
> - we will have to pay the price of the entry evaluation for every single
> read we do on any entry
Every single first read correct?
> #2 pros :
> - atomicity is somehow easier to enforce, but we still have some case where
> we have to update more than one entry when managing APs
> - potentially let the system in a consistent state if it crashes
Yeah this is making #2 look like a very poor option. I'd call it a con
rather than a pro but I think you meant this.
> #2 cons :
> - very complex to implement
> - there are some cases (like for the rename/move operations) where we either
> decide to impact a set of entries for nothing, or we have to immediately
> update a set of entries, defeating one of the [pros]
> - the first reader pay the price
Screw option #2 I see now why it's really worse than even #0.
> #3 pros :
> - the time it costs to update the entries is paid only once
> - update only the set of impacted entries
> #3 cons :
> - atomicity is not enforced, if the server crashes in the middle of an
> operation, it's unstable
> - it can take forever to update a huge base, assuming that we have to update
> all the entries, and it make take a lot of time at 1000 entries written per
> second
> - implementation is complex, but less than for #2
Inconsistency risk is something that we cannot take. #2 is the best
option or #0 I say. Better not to do if for now if we cannot do #2.
> At this point, I'm facing some big issue... If I want the rename/move
> operations implemented for option #2, it will imply either that we might
> impact many more entries than necessary, as I have to tell the parent AP
> that it's not up to date, *or* I have to adopt the #3 solution and update
> immediately the children entries...
>
> Ok, let's face the facts : maybe option #1 is the only possible solution we
> have atm, from different aspects :
> - delay : implementing option #1 is probably a matter of one week
Look if we make time a big deal and pump out a half assed solution
we're screw ourselves and our users. It will require more maintenance
and upkeep while not reaching our mark. I know you are of the same
mindset.
Forcing a fit right now is just going to result in something bad.
We're better off leaving this as is than trying to force fit something
that is not really that valuable. Why am I thinking this way?
(1) This is an optimization but we combined it with the extra IAP
feature which to date never got used.
(2) This is about administration which is rarely ever done.
(3) Until tooling appears to enable users these features are really
not going to hit mass utilization.
So why force something now that you yourself don't like the way we
have to compromise.
> - atomicity : it's the only solution that guarantees it
Yep, but we don't have to implement full atomicity but only the needed
scaffolding to be able to make the CL behave appropriately.
> The price to pay is a slower server for search operations, as each entry
> will be evaluated again and again.
>
> I don't know about you, but (running away from such things like "I feel in
> my guts", "I believe that", etc) from the technical point of view and the
> schedule point of view, I think that what matters right now are the
> following points :
Yes the gut is important and I say this sometimes too, but we have
force each other to figure out where that gut (subconscious) is
getting it's reasoning from. Not so easy.
> - we need to get the AP working fully, following the specifications and
> constraints
> - we need to get a 2.0-M1 out to be able to move to the next step
> - we need to implement a Transaction Manager in 2.0-M1 in order to implement
> a better strategy (ie #2 or #3)
>
> Really the problem now, and this is what I was trying to workaround with
> option #2, is the atomicity constraint. We need a transaction manager badly,
> and we need it for 2.0
Glad we're converging here.
> , no matter what, before we can implement either #2 or
> #3. And once we get this feature added in the server, then I do think that
> #3 is the best possible option.
Yes yes yes ! I totally agree.
Some things really cannot progress unless some supporting
infrastructure is laid down. The AP subsystem is exactly suffering
from this. Everything must evolve together. Otherwise we wind up
implementing a lot of forced solutions that are less than optimal with
hacks, work arounds and one offs. Things get more complex and brittle
causing more pain down the line.
Let's switch into a mode of cleaning up the shop after all this heavy
sawing and hammering. Sit down and plan a micro strategy with a clear
mind and clean shop. Then knock this out in a solid and easy way.
The AP thing was getting way too hard. I commend you on your effort,
and you're close but it's almost impossible to do without a local
transaction manager. Definitely impossible to do RIGHT without one.
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu