You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lecharny <el...@gmail.com> on 2010/05/21 02:45:28 UTC
Heads up about what's going on
Hi guys,
as many can check on the commits mailing list, a lot of
cleanup/refactoring is going on currently.
o Many PMD/checkstyle errors or warnings are being cleared by Felix, I
would like to thank him a lot for handling such a needed task. Please
feel free to answer his questions as he has not necessarily the clues
about the parts he is cleaning.
(Just one thing, Felix : it would be more than appreciated if you could
run the full test suite before committing, as some parts of the code is
not necessarily as easy to fix as it seems. I'm not blaming you for
that, I'm just trying to get the process to run more smoothly.)
o Stefan has moved around some modules, as they were just either empty,
or because their existence was questionnable
o Kiran is continuing his hard work to get the Master-Slave replication
working, and it's not easy.
o I have cleaned up a bit the JDBM BTree implementation, by adding
generics to it, removed some recursion, and adding back the tests that
were removed.
o Last, not least, I also have cleaned up the 'delete' operation which
is now slightly faster on my laptop (around 400 deletes per second), by
removing all the useless calls to lookups, list or normalization. The
impact is minimal (around 5 %) because most of the time is spent in the
backend updating indexes and the master table.
In the near future, we have some pretty heavy agenda :
- finish the master/slave replication system, and test it thouroughly
- decouple the BTree layer from the IO layer, so that we can have the
possibility to swap out this part
- finish the cleanup (PMD/CheckStyle)
- speedup the add/modify/modifyDN operations
- remove the brutal synchronization we hav in the backend and replace it
with something more convenient (either an optimistic concurrent locking,
or a MVCC system based BTree). We are still investigating this area atm.
That's pretty much it, from the insight I have, sorry if I missed something.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Heads up about what's going on
Posted by Alex Karasulu <ak...@apache.org>.
On Fri, May 21, 2010 at 3:45 AM, Emmanuel Lecharny <el...@gmail.com>wrote:
> Hi guys,
>
> as many can check on the commits mailing list, a lot of cleanup/refactoring
> is going on currently.
>
> o Many PMD/checkstyle errors or warnings are being cleared by Felix, I
> would like to thank him a lot for handling such a needed task. Please feel
> free to answer his questions as he has not necessarily the clues about the
> parts he is cleaning.
>
> (Just one thing, Felix : it would be more than appreciated if you could run
> the full test suite before committing, as some parts of the code is not
> necessarily as easy to fix as it seems. I'm not blaming you for that, I'm
> just trying to get the process to run more smoothly.)
>
> o Stefan has moved around some modules, as they were just either empty, or
> because their existence was questionnable
>
> o Kiran is continuing his hard work to get the Master-Slave replication
> working, and it's not easy.
>
> o I have cleaned up a bit the JDBM BTree implementation, by adding generics
> to it, removed some recursion, and adding back the tests that were removed.
>
> o Last, not least, I also have cleaned up the 'delete' operation which is
> now slightly faster on my laptop (around 400 deletes per second), by
> removing all the useless calls to lookups, list or normalization. The impact
> is minimal (around 5 %) because most of the time is spent in the backend
> updating indexes and the master table.
>
> In the near future, we have some pretty heavy agenda :
> - finish the master/slave replication system, and test it thouroughly
> - decouple the BTree layer from the IO layer, so that we can have the
> possibility to swap out this part
> - finish the cleanup (PMD/CheckStyle)
> - speedup the add/modify/modifyDN operations
> - remove the brutal synchronization we hav in the backend and replace it
> with something more convenient (either an optimistic concurrent locking, or
> a MVCC system based BTree). We are still investigating this area atm.
>
>
Great job guys! Really you're guys moving along very rapidly. I'm a bit
jealous these days for not being in the code with you guys but I'm mighty
proud to see y'all groking the hard stuff without breaking a sweat.
Hey save something for me to do OK :-D.
--
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu
Re: Heads up about what's going on
Posted by Felix Knecht <fe...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> That's an excellent tool. Just a question : is those tools running
> periodically or is this just on your computer ?
I setup a cronjob to generate and deploy the sites to
http://people.apache.org/~felixk/shared-docs/
http://people.apache.org/~felixk/apacheds-docs/
The job runs at 5 and 13 CEST
Regards
Felix
PS:
Let's see how long my machine works, no garantee ...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkv25B0ACgkQ2lZVCB08qHE+LgCgo/92R/t3jfskUHS3LidB44zz
rcEAnAlc4Y/6OWbGVES7zrkuzjc02pFf
=5ADe
-----END PGP SIGNATURE-----
Re: Heads up about what's going on
Posted by Felix Knecht <fe...@apache.org>.
> That's an excellent tool. Just a question : is those tools running
> periodically or is this just on your computer ?
I just generate the site (either locally or deploy it to
p.a.o/~felixk/...). This is the same site we had once on the oxylos
server deployed via CI build.
I can run the script on my machine and deploy it to p.a.o if this is
helpfull. Is twice a day enough (6am, 1pm)?
>
> In this case, can you share the commands you are using to generate them ?
cd
/home/felix/svn/apache/directory/trunk-with-dependencies
svn up /home/felix/svn/apache/directory/trunk-with-dependencies >
/home/felix/public_html/directory-reports.log
cd
shared
mvn clean site-deploy
-DdistMgmtSiteUrl=scpexe://localhost/home/felix/public_html/shared-docs
>>
/home/felix/public_html/directory-reports.log
cd
../apacheds
mvn clean site-deploy
-DdistMgmtSiteUrl=scpexe://localhost/home/felix/public_html/apacheds-docs >>
/home/felix/public_html/directory-reports.log
HTH
Felix
Re: Heads up about what's going on
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/21/10 10:20 AM, Felix Knecht wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>> It normally does... I don't why I got this Stack Overflow yesterday...
>>
> I got it today as well, no idea why I didn't got it yesterday...
> I'm sure it's my fault :(, thanks for fixing.
>
don't worry too much. Shit happens, we just have to fix it, whoever does
it :)
>
>> Yes, and that means we don't test enough in shared :/
>>
> We have a cobertura report for each module showing the test coverage.
> Some look better, some not.
>
That's an excellent tool. Just a question : is those tools running
periodically or is this just on your computer ?
In this case, can you share the commands you are using to generate them ?
Many thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Heads up about what's going on
Posted by Felix Knecht <fe...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> It normally does... I don't why I got this Stack Overflow yesterday...
I got it today as well, no idea why I didn't got it yesterday...
I'm sure it's my fault :(, thanks for fixing.
> Yes, and that means we don't test enough in shared :/
We have a cobertura report for each module showing the test coverage.
Some look better, some not.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkv2Ql4ACgkQ2lZVCB08qHH5TwCeNzRqXJ09ue03LLYuXF4jsdpQ
PyMAnjkvsUCzy3GMy5P8ipi3piWezcV6
=GHKR
-----END PGP SIGNATURE-----
Re: Heads up about what's going on
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/21/10 6:17 AM, Felix Knecht wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Thanks for heads up as well.
>
>
>> (Just one thing, Felix : it would be more than appreciated if you could
>> run the full test suite before committing, as some parts of the code is
>> not necessarily as easy to fix as it seems. I'm not blaming you for
>> that, I'm just trying to get the process to run more smoothly.)
>>
> I thought that "mvn clean install -Dintegration" will do this (and I try
> to run it before I commit the stuff, maybe I missed it on the latest
> commit, sorry).
It normally does... I don't why I got this Stack Overflow yesterday...
> And as I know not being fully fit on the code I really
> try to run the tests before doing the commit - and they passed for
> shared as wellas for apacheds. Is there a must using SunJDK instead
> JRokit or did I got the wrong command for running all the tests?
No. tests should pass with both of them. In any case, the pb was totally
JVM independant.
> Or is
> the test of apacheds also mandatory when doing any changes in shared?
Yes, and that means we don't test enough in shared :/
> It
> took for apacheds about 7 minutes 20 seconds - is this long enough?
>
IMHO, it's way to long, but it's hard to make it shorter ... That's a
real bummer to have to wait more than 10 minutes before committing, if
you want my opinion, but this is the only way to protect ourselves.
> BTW
> Some of the equals methods are still breaking the contract of symmetric,
> transitive [1] because of using reference comparison for collections
> [2], e.g. [3]. Is this wanted (there are more such cases).
>
> [1]
> http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Object.html#equals%28java.lang.Object%29
> [2]
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collection.html#equals%28java.lang.Object%29
> [3]
> http://people.apache.org/~felixk/shared-docs/xref/org/apache/directory/shared/ldap/aci/ProtectedItem.html#267
>
I don't know about protectedItem, I had no time yesterday to review this
piece of code. I must also admit this is not a part of the code I know...
FYI, I had to remove most of the ObjectClass.hashcode() containt to make
maven happy. I'm not pleased with the code I put instead, as it's a
gross approximation of what the hashcode should be, but anyway, it
should work.
There is one point whihi is important too when implementing hashcode()
methods : performances. In many cases, we don't want to compute those
hashcode everytime, this we store the result in a static variable. This
is cumbersome because it leads to either static initialization (not
always possible) or to some level of synchronization (sucks ...)
However, we have to keep this in mind, as it's really impact the server.
That being said, good job !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Heads up about what's going on
Posted by Stefan Seelmann <se...@apache.org>.
Felix Knecht schrieb:
> Thanks for heads up as well.
>
>> (Just one thing, Felix : it would be more than appreciated if you could
>> run the full test suite before committing, as some parts of the code is
>> not necessarily as easy to fix as it seems. I'm not blaming you for
>> that, I'm just trying to get the process to run more smoothly.)
>
> I thought that "mvn clean install -Dintegration" will do this (and I try
That is the right command.
Just a side note: the "integration" property is only used in the
syncrepl module to activate the integration tests profile. In core-integ
and server-integ the integration tests are activated by default. I'll
drop another mail about how to make this more consistent.
> to run it before I commit the stuff, maybe I missed it on the latest
> commit, sorry). And as I know not being fully fit on the code I really
> try to run the tests before doing the commit - and they passed for
> shared as wellas for apacheds. Is there a must using SunJDK instead
> JRokit or did I got the wrong command for running all the tests? Or is
IMO it is a good thing to run the test with different VMs.
> the test of apacheds also mandatory when doing any changes in shared? It
> took for apacheds about 7 minutes 20 seconds - is this long enough?
Yes, as apacheds heavily depends on shared. Additionally the LDAP API
depends on shared. I'd recommend to checkout trunk-with-dependencies [A]
as it includes all. On my laptop it takes 11 minutes 30 secons to build
trunk-with-dependencies.
>
> BTW
> Some of the equals methods are still breaking the contract of symmetric,
> transitive [1] because of using reference comparison for collections
> [2], e.g. [3]. Is this wanted (there are more such cases).
>
> [1]
> http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Object.html#equals%28java.lang.Object%29
> [2]
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collection.html#equals%28java.lang.Object%29
> [3]
> http://people.apache.org/~felixk/shared-docs/xref/org/apache/directory/shared/ldap/aci/ProtectedItem.html#267
Let me respond to you original mail.
Kind Regards,
Stefan
[A]http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependencies/
Re: Heads up about what's going on
Posted by Felix Knecht <fe...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Thanks for heads up as well.
> (Just one thing, Felix : it would be more than appreciated if you could
> run the full test suite before committing, as some parts of the code is
> not necessarily as easy to fix as it seems. I'm not blaming you for
> that, I'm just trying to get the process to run more smoothly.)
I thought that "mvn clean install -Dintegration" will do this (and I try
to run it before I commit the stuff, maybe I missed it on the latest
commit, sorry). And as I know not being fully fit on the code I really
try to run the tests before doing the commit - and they passed for
shared as wellas for apacheds. Is there a must using SunJDK instead
JRokit or did I got the wrong command for running all the tests? Or is
the test of apacheds also mandatory when doing any changes in shared? It
took for apacheds about 7 minutes 20 seconds - is this long enough?
BTW
Some of the equals methods are still breaking the contract of symmetric,
transitive [1] because of using reference comparison for collections
[2], e.g. [3]. Is this wanted (there are more such cases).
[1]
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Object.html#equals%28java.lang.Object%29
[2]
http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collection.html#equals%28java.lang.Object%29
[3]
http://people.apache.org/~felixk/shared-docs/xref/org/apache/directory/shared/ldap/aci/ProtectedItem.html#267
Thanks for help
Felix
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkv2CUEACgkQ2lZVCB08qHHblQCffUT9sAS+mPJsH5Jb9Qqj/kZ+
tygAmwYyHWBOS6PgBUqL13eYBy9IN+Wo
=aar7
-----END PGP SIGNATURE-----