You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jean-Philippe Robichaud <Je...@nuance.com> on 2007/04/05 03:54:08 UTC
Lucene and Javolution: A good mix ?
Hello Dear Lucene coders!
Some of you may remember, I'm using lucene for a product (and many other
internal utilities). I'm also using another open source library called
Javolution (www.javolution.org <http://www.javolution.org/> ) which does
many things, one of them being to offer excellent replacements for
ArrayList/Map/... and a super good memory management extension to the
java language.
As I'm [trying to] follow the conversations on this list, I see that
many of you are working towards optimizing lucene in term of memory
footprint and speed. I just finished optimizing my code (not lucene
itself, but my code written on top of it) using Javolution PoolContext
and the FastList/FastMap/... classes. The resulting speedup is a 6
times faster code.
Javolution make it easy to recycle objects and do some object allocation
on the stack rather than on the head, which remove stress on the garbage
collector. Javolution also offers 2 classes (Text and TextBuilder) to
replace String/StringBuffer which are perfect for anything related to
string manipulation and some C "union/struct" equivalent for java. The
thing is really great.
Would anyone be interested in doing Lucene a face lift and start using
javolution as a core lucene dependency? I understand that right now,
lucene is free of any dependencies, which is quite great, but anyone
interested in doing fast/lean/stable java application should seriously
consider using javolution anyway.
Any thoughts?
Jp
Re: Lucene and Javolution: A good mix ?
Posted by Mike Klaas <mi...@gmail.com>.
On 4/4/07, Jean-Philippe Robichaud <Je...@nuance.com> wrote:
> I understand your concerns!
>
> I was a little skeptical at the beginning. But even with the 1.5 jvm,
> the improvements still holds.
>
> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
> index time or query time. While the new garbage collector strategies did
> seriously improve since java 1.4, the gains are still there as the
> object "creation" is also a cost that javolution easily saves us from.
I think the best approach at convincing people would be to produce a
patch that implements some of the suggested changes, and benchmark it.
As it stands, the positives are all hypothetical and the negatives
rather tangible.
-MIke
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Lucene and Javolution: A good mix ?
Posted by Jean-Philippe Robichaud <Je...@nuance.com>.
I understand your concerns!
I was a little skeptical at the beginning. But even with the 1.5 jvm,
the improvements still holds.
Lucene creates a lots of "garbage" (strings, tokens, ...) either at
index time or query time. While the new garbage collector strategies did
seriously improve since java 1.4, the gains are still there as the
object "creation" is also a cost that javolution easily saves us from.
What javolution requires to give the best is can is for you to make
certain critical classes extends the RealtimeObject class and implement
a Factory pattern inside. Once this is done, you can now fully profit
from the Javolution features. Even without doing that, we could still
benefit from the FastList/FastMap classes being already pooled and the
possibility to 'thread-safely' iterate lists/maps without creating any
objects.
Javolution is also released for gcj, which is great since it won't
interfere with the lucene's gcj effort.
>From what I can foresee, the pros/cons would be:
Pros:
Leaner memory footprint
Saving many cpu cycles
Cons:
Adding a dependency to lucene codebase
Lucene developers must get familiar with the "Context" concepts
Jp
-----Original Message-----
From: robert engels [mailto:rengels@ix.netcom.com]
Sent: Wednesday, April 04, 2007 10:31 PM
To: java-dev@lucene.apache.org
Subject: Re: Lucene and Javolution: A good mix ?
I would suggest that the Javolution folks do their tests against
modern JVM...
I have followed the Javolution project for some time, and while I
agree that some of the techniques should improve things, I think that
modern JVMs do most of this work for you (and the latest class
libraries also help - StringBuilder and others).
I also think that when you start doing you own memory management you
might as well write the code in C/C++ because you need to use similar
techniques (similar to the resource management when using SWT).
Just my thoughts.
On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote:
> Hello Dear Lucene coders!
>
>
>
> Some of you may remember, I'm using lucene for a product (and many
> other
> internal utilities). I'm also using another open source library
> called
> Javolution (www.javolution.org <http://www.javolution.org/> ) which
> does
> many things, one of them being to offer excellent replacements for
> ArrayList/Map/... and a super good memory management extension to the
> java language.
>
>
>
> As I'm [trying to] follow the conversations on this list, I see that
> many of you are working towards optimizing lucene in term of memory
> footprint and speed. I just finished optimizing my code (not lucene
> itself, but my code written on top of it) using Javolution PoolContext
> and the FastList/FastMap/... classes. The resulting speedup is a 6
> times faster code.
>
>
>
> Javolution make it easy to recycle objects and do some object
> allocation
> on the stack rather than on the head, which remove stress on the
> garbage
> collector. Javolution also offers 2 classes (Text and
> TextBuilder) to
> replace String/StringBuffer which are perfect for anything related to
> string manipulation and some C "union/struct" equivalent for java.
> The
> thing is really great.
>
>
>
> Would anyone be interested in doing Lucene a face lift and start using
> javolution as a core lucene dependency? I understand that right now,
> lucene is free of any dependencies, which is quite great, but anyone
> interested in doing fast/lean/stable java application should seriously
> consider using javolution anyway.
>
>
>
> Any thoughts?
>
>
>
> Jp
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene and Javolution: A good mix ?
Posted by robert engels <re...@ix.netcom.com>.
I would suggest that the Javolution folks do their tests against
modern JVM...
I have followed the Javolution project for some time, and while I
agree that some of the techniques should improve things, I think that
modern JVMs do most of this work for you (and the latest class
libraries also help - StringBuilder and others).
I also think that when you start doing you own memory management you
might as well write the code in C/C++ because you need to use similar
techniques (similar to the resource management when using SWT).
Just my thoughts.
On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote:
> Hello Dear Lucene coders!
>
>
>
> Some of you may remember, I'm using lucene for a product (and many
> other
> internal utilities). I'm also using another open source library
> called
> Javolution (www.javolution.org <http://www.javolution.org/> ) which
> does
> many things, one of them being to offer excellent replacements for
> ArrayList/Map/... and a super good memory management extension to the
> java language.
>
>
>
> As I'm [trying to] follow the conversations on this list, I see that
> many of you are working towards optimizing lucene in term of memory
> footprint and speed. I just finished optimizing my code (not lucene
> itself, but my code written on top of it) using Javolution PoolContext
> and the FastList/FastMap/... classes. The resulting speedup is a 6
> times faster code.
>
>
>
> Javolution make it easy to recycle objects and do some object
> allocation
> on the stack rather than on the head, which remove stress on the
> garbage
> collector. Javolution also offers 2 classes (Text and
> TextBuilder) to
> replace String/StringBuffer which are perfect for anything related to
> string manipulation and some C "union/struct" equivalent for java.
> The
> thing is really great.
>
>
>
> Would anyone be interested in doing Lucene a face lift and start using
> javolution as a core lucene dependency? I understand that right now,
> lucene is free of any dependencies, which is quite great, but anyone
> interested in doing fast/lean/stable java application should seriously
> consider using javolution anyway.
>
>
>
> Any thoughts?
>
>
>
> Jp
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org