You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cam Bazz <ca...@gmail.com> on 2008/01/15 13:17:49 UTC
lucene as a graph store
Hello;
I like to use lucene as a graph store. The graph representation is a list of
edges. Consider the code below:
final int commitCount = 16 * 1024;
final int numObj = 1024 * 1024;
Analyzer analyzer = new KeywordAnalyzer();
FSDirectory directory = FSDirectory.getDirectory("c:\\LuceneAdd");
IndexWriter writer = new IndexWriter(directory, analyzer, true);
Document doc;
long start = System.currentTimeMillis();
Random r = new Random(System.currentTimeMillis());
for(int i=0; i<numObj; i++) {
doc = new Document();
doc.add(new Field("srcKey", NumberTools.longToString(i),
Field.Store.YES, Field.Index.UN_TOKENIZED));
doc.add(new Field("dstKey",
NumberTools.longToString(r.nextInt(numObj)),
Field.Store.YES, Field.Index.UN_TOKENIZED));
doc.add(new Field("linkKey",
NumberTools.longToString(r.nextInt(16)),
Field.Store.YES, Field.Index.UN_TOKENIZED));
doc.add(new Field("linkValue", NumberTools.longToString(
r.nextInt(256)), Field.Store.YES, Field.Index.UN_TOKENIZED));
writer.addDocument(doc);
if(i%commitCount==0) {
long now = System.currentTimeMillis();
System.out.println(i + ":" + (now-start));
start = now;
}
}
writer.optimize();
writer.close();
directory.close();
Basically I am adding a large number of documents from srcKey = i to dstKey
= random and two other string fields - linkKey and linkValue.
Compared to a normal database store, or an oodbms such as perst or db4o -
lucene takes longer to index.
However, it is much faster in searching, finding, retrieving records.
I can make 16384 random lookups over 1Million entries in 0.8 seconds. This
is excellent time. (I have been benchmarking for a long time)
Typically, when number of objects in BTree based structure in an oodbms for
example increase, the search and add times also increase.
Will lucene have the same problem and how can I overcome it if it does.
Looking at the above code - does anyone has any recomendations
to improve index performance. (also what can I do to improve search
performance)
While searching with an indexsearcher - does lucene do any caching? usually
MRU caches are used to accomplish this.
Any ideas,help,recomendations greatly appreciated.
Best,
-C.B.
Re: lucene as a graph store
Posted by Cam Bazz <ca...@gmail.com>.
Usually for implementing things like page rank, or doing centrality metric
calculations or maybe dijkstras shortest term, this kind of (list of edges)
graph is not best at performance.
I like to use lucene for simple operations like neighboors of this node, or
2 degree neighboors of this node.
is updating an index costly operation in lucene? I dont think there will too
much updates, but rather deletes.
when I delete documents from an index what things should I be careful about.
Best.
-C.B.
On Jan 15, 2008 3:31 PM, Grant Ingersoll <gs...@apache.org> wrote:
> I guess the question comes down to what kind of things are you going
> to do w/ this graph? How often are you updating links, etc? I can't
> say Lucene was designed for this kind of thing, but I am constantly
> amazed at what people use Lucene for, so I won't say it can't be
> done. I don't know how efficient it would be for doing things like
> PageRank or other graph algorithms, but I would be interesting in
> hearing more about what you have in mind.
>
> Lucene doesn't do much caching at the document level, but that is
> fairly easy to implement, but it does bring some terms, etc. into
> memory, and you may have a look at the FieldCache.
>
> -Grant
>
> On Jan 15, 2008, at 7:17 AM, Cam Bazz wrote:
>
> > Hello;
> >
> > I like to use lucene as a graph store. The graph representation is a
> > list of
> > edges. Consider the code below:
> >
> > final int commitCount = 16 * 1024;
> > final int numObj = 1024 * 1024;
> >
> > Analyzer analyzer = new KeywordAnalyzer();
> > FSDirectory directory = FSDirectory.getDirectory("c:\
> > \LuceneAdd");
> > IndexWriter writer = new IndexWriter(directory, analyzer,
> > true);
> >
> > Document doc;
> > long start = System.currentTimeMillis();
> >
> > Random r = new Random(System.currentTimeMillis());
> >
> > for(int i=0; i<numObj; i++) {
> > doc = new Document();
> > doc.add(new Field("srcKey", NumberTools.longToString(i),
> > Field.Store.YES, Field.Index.UN_TOKENIZED));
> > doc.add(new Field("dstKey",
> > NumberTools.longToString(r.nextInt(numObj)),
> > Field.Store.YES, Field.Index.UN_TOKENIZED));
> > doc.add(new Field("linkKey",
> > NumberTools.longToString(r.nextInt(16)),
> > Field.Store.YES, Field.Index.UN_TOKENIZED));
> > doc.add(new Field("linkValue", NumberTools.longToString(
> > r.nextInt(256)), Field.Store.YES, Field.Index.UN_TOKENIZED));
> >
> > writer.addDocument(doc);
> >
> > if(i%commitCount==0) {
> > long now = System.currentTimeMillis();
> > System.out.println(i + ":" + (now-start));
> > start = now;
> > }
> >
> > }
> >
> > writer.optimize();
> > writer.close();
> > directory.close();
> >
> >
> > Basically I am adding a large number of documents from srcKey = i to
> > dstKey
> > = random and two other string fields - linkKey and linkValue.
> >
> > Compared to a normal database store, or an oodbms such as perst or
> > db4o -
> > lucene takes longer to index.
> > However, it is much faster in searching, finding, retrieving records.
> >
> > I can make 16384 random lookups over 1Million entries in 0.8
> > seconds. This
> > is excellent time. (I have been benchmarking for a long time)
> >
> > Typically, when number of objects in BTree based structure in an
> > oodbms for
> > example increase, the search and add times also increase.
> >
> > Will lucene have the same problem and how can I overcome it if it
> > does.
> > Looking at the above code - does anyone has any recomendations
> > to improve index performance. (also what can I do to improve search
> > performance)
> >
> > While searching with an indexsearcher - does lucene do any caching?
> > usually
> > MRU caches are used to accomplish this.
> >
> > Any ideas,help,recomendations greatly appreciated.
> >
> > Best,
> > -C.B.
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
> http://www.lucenebootcamp.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: lucene as a graph store
Posted by Grant Ingersoll <gs...@apache.org>.
I guess the question comes down to what kind of things are you going
to do w/ this graph? How often are you updating links, etc? I can't
say Lucene was designed for this kind of thing, but I am constantly
amazed at what people use Lucene for, so I won't say it can't be
done. I don't know how efficient it would be for doing things like
PageRank or other graph algorithms, but I would be interesting in
hearing more about what you have in mind.
Lucene doesn't do much caching at the document level, but that is
fairly easy to implement, but it does bring some terms, etc. into
memory, and you may have a look at the FieldCache.
-Grant
On Jan 15, 2008, at 7:17 AM, Cam Bazz wrote:
> Hello;
>
> I like to use lucene as a graph store. The graph representation is a
> list of
> edges. Consider the code below:
>
> final int commitCount = 16 * 1024;
> final int numObj = 1024 * 1024;
>
> Analyzer analyzer = new KeywordAnalyzer();
> FSDirectory directory = FSDirectory.getDirectory("c:\
> \LuceneAdd");
> IndexWriter writer = new IndexWriter(directory, analyzer,
> true);
>
> Document doc;
> long start = System.currentTimeMillis();
>
> Random r = new Random(System.currentTimeMillis());
>
> for(int i=0; i<numObj; i++) {
> doc = new Document();
> doc.add(new Field("srcKey", NumberTools.longToString(i),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
> doc.add(new Field("dstKey",
> NumberTools.longToString(r.nextInt(numObj)),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
> doc.add(new Field("linkKey",
> NumberTools.longToString(r.nextInt(16)),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
> doc.add(new Field("linkValue", NumberTools.longToString(
> r.nextInt(256)), Field.Store.YES, Field.Index.UN_TOKENIZED));
>
> writer.addDocument(doc);
>
> if(i%commitCount==0) {
> long now = System.currentTimeMillis();
> System.out.println(i + ":" + (now-start));
> start = now;
> }
>
> }
>
> writer.optimize();
> writer.close();
> directory.close();
>
>
> Basically I am adding a large number of documents from srcKey = i to
> dstKey
> = random and two other string fields - linkKey and linkValue.
>
> Compared to a normal database store, or an oodbms such as perst or
> db4o -
> lucene takes longer to index.
> However, it is much faster in searching, finding, retrieving records.
>
> I can make 16384 random lookups over 1Million entries in 0.8
> seconds. This
> is excellent time. (I have been benchmarking for a long time)
>
> Typically, when number of objects in BTree based structure in an
> oodbms for
> example increase, the search and add times also increase.
>
> Will lucene have the same problem and how can I overcome it if it
> does.
> Looking at the above code - does anyone has any recomendations
> to improve index performance. (also what can I do to improve search
> performance)
>
> While searching with an indexsearcher - does lucene do any caching?
> usually
> MRU caches are used to accomplish this.
>
> Any ideas,help,recomendations greatly appreciated.
>
> Best,
> -C.B.
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene as a graph store
Posted by Karl Wettin <ka...@gmail.com>.
15 jan 2008 kl. 13.17 skrev Cam Bazz:
>
> Typically, when number of objects in BTree based structure in an
> oodbms for
> example increase, the search and add times also increase.
>
> Will lucene have the same problem and how can I overcome it if it
> does.
There is a benchmark package in the contrib section:
http://svn.apache.org/repos/asf/lucene/java/trunk/contrib/benchmark/
Implement a DocMaker and simulate your desired system.
> Looking at the above code - does anyone has any recomendations
> to improve index performance. (also what can I do to improve search
> performance)
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
> While searching with an indexsearcher - does lucene do any caching?
> usually
> MRU caches are used to accomplish this.
I have no idea, someone else have to help you out there. Perhaps the
Solr wiki have something about that? Also try searching the forum
archives, using Nabble for instance.
--
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org