Happy 2020 !

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

firs, I'd like to wish you all, users and dev, a very happy 2020 !


2019 was quite slow, for many reasons. I can't talk about others, but on 
my side, I had a big change in my day job just sucked all my free time, 
and made me vary hard to participate since august. Things are cooling 
down now (changing for a new job is always very taxing in term of free 
time).

Since august, I spent time on Mavibot, implementing transaction support. 
It was pretty hard, with some concurrent issues that took me one month 
to get fixed (mainly because I wasn't able to dedicate more than a few 
hours per week on it). It's not yet committed, but it works. The biggest 
plus is that when you activate a transaction, and make more than one 
update on a B-tree, all being done in memory up to the final commit, 
it's way, way faster. It more or less double the number of updates per 
second when you double the number of updates per transaction (well, 
obviously less than double ;-). There are reasons for this speedup :

- the updated B-tree get flushed on disk only once, we then save many 
pages writes (typically, the root page get flushed only once per txn, 
instead of being flushed for every update.

- technical B-trees (CopiedPages and BtreeOfBtrees) and recordManager 
header get flushed only once too

There are some other improvements too: I have replaced the BtreeOfBtrees 
by a ListOfBtrees. It's way more dense, as it only keeps the latest 
offset of each existing B-tree. That also means we don't write an extra 
B-tree when we commit a transaction. The drawback is that we have to 
write the full list after each commit, but we are talking about only a 
few pages (each page containing up to 500 B-tree offset for a 4Kb page). 
The second drawback is that of we have a brutal crash, we may lose some 
old B-tree references if they were used by long lasting transactions. 
Note that we still keep a track of all the used B-tree reference in 
memory until they get removed.

The free pages management is not active at this point, and this is the 
next step to be completed. I want to change the existing algorithm to 
something better. Currently, we only clean unused pages when the oldest 
active read transaction is released, which is an issue if we have very 
long lasting read transactions: we may have a growing database that is 
never cleaned up if the transaction is never released. The alternative 
algorithm uses the CopiedPageBtree structure that keep a track of copied 
pages when a write transaction has been committed. It allows us to get 
rid of pages from in-between read transaction when they get released, 
assuming those pages have been copied. The algorithm is slighly more 
complex to implement, but guarantee that long lasting transactions won't 
clog the database.


Otherwise, a LDAP API release has been done, the first GA. This is quite 
a milestone ! There is still a lot of work to do, and that includes the 
documentation, which is lagging.


On ApacheDS front, this is a bit more complicated. I have triaged all 
the issues, adding each of them inti some categories, and started to 
work on the PasswordPolicies issues (there are many). It takes a lot of 
time though...

Here is what I intend to do then :

- cut a release as is,  in order to have a code base that use teh 2.0 
LDAP API. Some people are waiting for month to have a test envirronement 
that contains some major fixes, so regardless on where I'm on PP fixes, 
I think it's urgent to have this release done.

- fix the packaging. We need to have a Mac OSX package that works, and 
that means something signed (the new Apple policy for installable 
package is a bit strict...). I have this signature available, I just 
need to get it applied to the package. We also need a Windows 64 
package. Currently, we use a 32bits library - NSIS - which is not 
maintained anymore. I'm looking into YAJSW, which seems quite 
compatible. Then we need to get Mavibot upgraded with the version i'm 
working on, and make it the default database, to solve the long lasting 
database corruption that bog is for years...

Plus many other bugs to work on !

That means we will have many releases before reaching a GA :/

At this point, I may need some hands !


Regarding Studio, I think Stefan is waiting for ApacheDS to be released 
to cut a big fix release. Stefan, feel free to comment.


In any case, we need some help in all the aspects of the project 
(ApacheDS, LDAP API, Fortress,Studio, etc), and that include doculmentation.

I haven't mentioned Docker, but we would appreciate any effort to 
containerize ApacheDS, Fortress and Studio. This is not complex, Shawn 
did it for Fortres, but it would be cool to have it documented. I wonder 
if it's not the proper approach compared to creating packages for 
various platforms...


Anyway, lots of work on my plate, but happy to do it !

Feel free to comment, fo course !


Thanks

Emmanuel


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org

Re: Happy 2020 !

[Shawn McKinney <sm...@apache.org>]

Happy new year!

> On Jan 3, 2020, at 9:19 AM, Emmanuel Lécharny <el...@gmail.com> wrote:
> 
> In any case, we need some help in all the aspects of the project (ApacheDS, LDAP API, Fortress,Studio, etc), and that include doculmentation.
> 
> I haven't mentioned Docker, but we would appreciate any effort to containerize ApacheDS, Fortress and Studio. This is not complex, Shawn did it for Fortres, but it would be cool to have it documented. I wonder if it's not the proper approach compared to creating packages for various platforms…
> 

It was Stefan who containerized apacheds (and openldap) for Fortress’ Jenkins tests.

I’m not one to jump on bandwagons, like Docker (and I know you’re not either), but this is how infra gets deployed.  (Nobody installs anything anymore)

> Anyway, lots of work on my plate, but happy to do it !
> 
> Feel free to comment, fo course !

Echo sentiments, lots of todos, minimal time available in order to get it done means we have to choose wisely.  I appreciate the efforts of this project’s volunteers. 

—
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org