You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xindice-users@xml.apache.org by Beni Ruef <br...@es.unizh.ch> on 2002/11/10 15:50:45 UTC
Performance question (Am I doing something wrong?)
I just installed Xindice 1.0 on my iBook under Mac OS X (with enough
RAM, i.e. a reasonably fast machine).
The (TEI encoded) texts I'm interested in look like this:
.
.
<s n="id-1.1"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
pos="AT1">a</w><w pos="NN1">sentence</w><c pos="YSTP">.</c></s>
<s n="id-1.2"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
pos="DD1">another</w><w pos="PN1">one</w><c pos="YSTP">.</c></s>
</body></text>
</TEI.2>
and the simplest queries like this:
xindice xpath -c /db/myCollection -q
'/TEI.2/text/body/s[w="another"]'
With a test corpus containing three documents and a total of ca. 700
KBytes the above query takes ca. 9 seconds...
More surprisingly, even a simple retrieval (xindice rd) of a 200K
document needs 5 seconds!
After having run
xindiceadmin ai -c /db/myCollection -p w -n wordform
things improve slightly as the same query takes now 4.5 seconds, a query
with a non-existing word form "only" 2.5 seconds. BTW, there seems to
be an overhead of ca. 2 seconds as any operation takes at least 2
seconds...
Obviously, this is still way too slow to be usable as I'm planning to
work with corpora containing some 100 million words...
So what am I doing (terribly ;-) wrong and what can be improved?! What
about this overhead and what about the indexer switches like pagesize?
Thanks in advance, Cheers
-Beni
cpu vs. i/o times (was: Re: Performance question (Am I doing something wrong?))
Posted by Beni Ruef <br...@es.unizh.ch>.
On Monday, November 11, 2002, at 03:38, Jeff Greif wrote:
>> % /usr/bin/time xindice xpath -c /db/resumes -q
>> "/candidate[biographic_data/id='ANON2021']" >ANON2021_2.xml
>> 372.72 real 2.22 user 1.22 sys
>
> Notice that the cpu time is only
> about 3s, so all the delay is in some kind of i/o or paging...
Notice that this is just the cpu time of the xindice client, i.e. the
command line interface! If you're running your server on the same host
you'll notice (with top or ps) that the "remaining" seconds are eaten by
the server process(es).
-Beni
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
If I understand, you have an 80MB collection. Even if xindice were reading
this 1 disk block at a time to scan the entire collection in some dumb
fashion, it should be faster than 256s. Notice that the cpu time is only
about 3s, so all the delay is in some kind of i/o or paging -- something
involving how the operating system is configured, probably. Sorry to be of
so little help.
Jeff
----- Original Message -----
From: "Matthew Van Horn" <ho...@accesstech.com>
To: <xi...@xml.apache.org>
Sent: Sunday, November 10, 2002 10:33 PM
Subject: Re: Performance question (Am I doing something wrong?)
> Contrary to what you might think, Jeff, the performance actually
> degrades a bit when I am explicit:
> (By the way, "candidate" is the root element and there is only one
> document with this ID)
>
> % /usr/bin/time xindice xpath -c /db/resumes -q
> "/candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
> 372.72 real 2.22 user 1.22 sys
> % /usr/bin/time xindice xpath -c /db/resumes -q
> "//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
> 334.25 real 2.28 user 1.09 sys
>
> I think the above times are both substantially slower because this
> time I had JBuilder running at the same time.
>
> When I shut down JBuilder, I got the following, which still shows the
> "//candidate" is faster than "/candidate"
>
> % /usr/bin/time xindice xpath -c /db/resumes -q
> "/candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
> 256.52 real 2.10 user 1.12 sys
> % /usr/bin/time xindice xpath -c /db/resumes -q
> "//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
> 171.18 real 1.92 user 0.87 sys
>
> Maybe Xindice is optimized for //element searching someow?
Re: Performance question (Am I doing something wrong?)
Posted by Matthew Van Horn <ho...@accesstech.com>.
Contrary to what you might think, Jeff, the performance actually
degrades a bit when I am explicit:
(By the way, "candidate" is the root element and there is only one
document with this ID)
% /usr/bin/time xindice xpath -c /db/resumes -q
"/candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
372.72 real 2.22 user 1.22 sys
% /usr/bin/time xindice xpath -c /db/resumes -q
"//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
334.25 real 2.28 user 1.09 sys
I think the above times are both substantially slower because this
time I had JBuilder running at the same time.
When I shut down JBuilder, I got the following, which still shows the
"//candidate" is faster than "/candidate"
% /usr/bin/time xindice xpath -c /db/resumes -q
"/candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
256.52 real 2.10 user 1.12 sys
% /usr/bin/time xindice xpath -c /db/resumes -q
"//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
171.18 real 1.92 user 0.87 sys
Maybe Xindice is optimized for //element searching someow?
On Monday, November 11, 2002, at 02:34 PM, Jeff Greif wrote:
> Get rid of the descendant part of the query!
> Your index on id can tell you which documents have the id requested,
> but
> there is still a search through each of those documents for
> //candidate,
> which must examine every path through the document. If candidate
> appears in
> just one position in each document, specify the ancestor nodes
> explicitly.
> If it appears in a few places, specify disjunctions at the explicit
> ancestor
> nodes.
> Your Xpath query should thus look like
> /foo/bar/candidate[biographic_data/id='...']
> or perhaps
> /foo/bar/candidate[biographic_data/id='...'] or
> /foo/quux/zork/candidate[biographic_data/id='...']
>
> Jeff
> ----- Original Message -----
> From: "Matthew Van Horn" <ho...@accesstech.com>
> To: <xi...@xml.apache.org>
> Sent: Sunday, November 10, 2002 5:52 PM
> Subject: Re: Performance question (Am I doing something wrong?)
>
>
>>
>> On Monday, November 11, 2002, at 04:45 AM, Mark J. Stang wrote:
>>
>>> Not wrong. If that is what your tests show, then you are not doing
>>> anything wrong. Not if your test is "how long will a command-line
>>> query take?" The question is, is that your entire application and
>>
>>> I had a problem with Mac OS X with every query taking
>>> forever. Turned out that the database initialization was being
>>> done every time and it took forever. Ran the same program
>>> on a single processor Mac and a dual-processor Mac. One
>>> was fast the other slow.
>>
>> I am also getting extremely slow query performance in OS X (10.2). I
>> have a collection with about 8000 documents, each of which is about
>> 8-10K. The following are the times for retrieving a document, and
>> getting the same document via a query. Since I would like to be able
>> to
>> use the command line often at this stage of the project, is there any
>> way to improve this performance? (BTW these times are with an index on
>> the id element.)
>>
>> % /usr/bin/time xindice rd -c /db/resumes -n ANON2021 > ANON2021.xml
>> 2.41 real 1.66 user 0.37 sys
>>
>> % /usr/bin/time xindice xpath -c /db/resumes -q
>> "//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
>> 234.44 real 2.09 user 0.90 sys
>>
>>
>
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
Get rid of the descendant part of the query!
Your index on id can tell you which documents have the id requested, but
there is still a search through each of those documents for //candidate,
which must examine every path through the document. If candidate appears in
just one position in each document, specify the ancestor nodes explicitly.
If it appears in a few places, specify disjunctions at the explicit ancestor
nodes.
Your Xpath query should thus look like
/foo/bar/candidate[biographic_data/id='...']
or perhaps
/foo/bar/candidate[biographic_data/id='...'] or
/foo/quux/zork/candidate[biographic_data/id='...']
Jeff
----- Original Message -----
From: "Matthew Van Horn" <ho...@accesstech.com>
To: <xi...@xml.apache.org>
Sent: Sunday, November 10, 2002 5:52 PM
Subject: Re: Performance question (Am I doing something wrong?)
>
> On Monday, November 11, 2002, at 04:45 AM, Mark J. Stang wrote:
>
> > Not wrong. If that is what your tests show, then you are not doing
> > anything wrong. Not if your test is "how long will a command-line
> > query take?" The question is, is that your entire application and
>
> > I had a problem with Mac OS X with every query taking
> > forever. Turned out that the database initialization was being
> > done every time and it took forever. Ran the same program
> > on a single processor Mac and a dual-processor Mac. One
> > was fast the other slow.
>
> I am also getting extremely slow query performance in OS X (10.2). I
> have a collection with about 8000 documents, each of which is about
> 8-10K. The following are the times for retrieving a document, and
> getting the same document via a query. Since I would like to be able to
> use the command line often at this stage of the project, is there any
> way to improve this performance? (BTW these times are with an index on
> the id element.)
>
> % /usr/bin/time xindice rd -c /db/resumes -n ANON2021 > ANON2021.xml
> 2.41 real 1.66 user 0.37 sys
>
> % /usr/bin/time xindice xpath -c /db/resumes -q
> "//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
> 234.44 real 2.09 user 0.90 sys
>
>
Re: Performance question (Am I doing something wrong?)
Posted by Matthew Van Horn <ho...@accesstech.com>.
On Monday, November 11, 2002, at 04:45 AM, Mark J. Stang wrote:
> Not wrong. If that is what your tests show, then you are not doing
> anything wrong. Not if your test is "how long will a command-line
> query take?" The question is, is that your entire application and
> I had a problem with Mac OS X with every query taking
> forever. Turned out that the database initialization was being
> done every time and it took forever. Ran the same program
> on a single processor Mac and a dual-processor Mac. One
> was fast the other slow.
I am also getting extremely slow query performance in OS X (10.2). I
have a collection with about 8000 documents, each of which is about
8-10K. The following are the times for retrieving a document, and
getting the same document via a query. Since I would like to be able to
use the command line often at this stage of the project, is there any
way to improve this performance? (BTW these times are with an index on
the id element.)
% /usr/bin/time xindice rd -c /db/resumes -n ANON2021 > ANON2021.xml
2.41 real 1.66 user 0.37 sys
% /usr/bin/time xindice xpath -c /db/resumes -q
"//candidate[biographic_data/id='ANON2021']" > ANON2021_2.xml
234.44 real 2.09 user 0.90 sys
Re: Performance question (Am I doing something wrong?)
Posted by "Mark J. Stang" <ma...@earthlink.net>.
To cache collections, create a Singleton. Inside the
singleton store off the databasemgr and collections. Reuse
the collections!
HTH,
Mark
Beni Ruef wrote:
> On Sunday, November 10, 2002, at 08:45, Mark J. Stang wrote:
>
> > Not wrong. If that is what your tests show, then you are not doing
> > anything wrong. Not if your test is "how long will a command-line
> > query take?"
>
> Got your point, excuse my stupidity ;-)
>
> > Try adding in a thousand such documents and
> > index them. My guess is that the query time will increase but
> > it will be hard to measure the increase.
>
> That's correct but there's a big but: searching for rare words (as my
> example was) is indeed rather independent of the corpus size but
> searching for more common words is another story (the following numbers
> are with a 30 MB corpus):
> Searching for a word occuring only twice takes ca. 3 seconds, searching
> for a word occuring 13 times takes 70 seconds, and a word occuring 1300
> times takes even more than 6 minutes!
>
> This is a typical characteristic of linguistic corpora: although the
> number of words is virtually unlimited the number of unique word forms
> is quite restricted: one of my corpora has 106'000 words but only 19'000
> different word forms. It's even worse with word categories or POS tags
> (the 'pos' attributes in my example) as there are just 50 - 100 of
> them. Technically speaking, the values used as keys in the index are
> far from unique -- I don't know how Xindice can handle this problem.
>
> > The only accurate way to measure the time is to build a small
> > program and test the actual call to the database. Try using
> > Example1.java, that one has always worked well for me.
>
> I did this and it takes two seconds (measured between getCollection()
> and getIterator()) for my "rare word" query.
>
> > I had a problem with Mac OS X with every query taking
> > forever. Turned out that the database initialization was being
> > done every time and it took forever.
>
> But why, isn't the Java code identical? (Disclaimer: I know nothing
> about Java ;-)
>
> > The same query was faster on
> > Windows and Linux.
>
> I did my benchmarks on a slow Linux machine (AMD-K6 350 MHz) and they
> take only twice as long as on my (fast :-) iBook.
>
> > I ended up caching my collections.
>
> How do you do this?
>
> > This is from an e-mail dated September 5, 2001.
> >
> > Kimbro Staken wrote:
> >
> >> computer: 750MHZ P3 256MB RAM Laptop running Mandrake Linux 8
> >> jdk: Sun 1.3.0_04
> >> Dataset size: 149,025 documents 601MB
> >> Insertion time (no indexes): 1 hour 45 minutes which is roughly 1,424
> >> docs
> >> per minute or 24 per second.
> >> Collection size: 657MB
> >> Document retrieval: 2 seconds (including VM startup which is most of
> >> the
> >> time)
> >> Full collection scan query /disc[id = '11041c03']: 12 minutes
> >> Index creation: 13.5 minutes
> >> Index based query /disc[id = '11041c03']: 2.12 seconds (including VM
> >> startup which is most of that time)
>
> I'd really like such a fast response :-) but these are unique keys...
>
> Cheers
> -Beni
--
Mark J Stang
System Architect
Cybershop Systems
Re: Performance question (Am I doing something wrong?)
Posted by "Mark J. Stang" <ma...@earthlink.net>.
Only register the database once and reuse your collections.
hth,
Mark
Matthew Van Horn wrote:
> On Tuesday, November 12, 2002, at 11:47 AM, Jeff Greif wrote:
>
> > I think I see the problem. I believe (if I've read the source code
> > correctly) that in xindice, your index on <w> is essentially a map from
> > values of <w> to document keys.
> > ...snippage...
> > Clearly, if I'm not confused, xindice is optimized for small documents
> > with
> > not much repeating structure, and its indexing mechanism is not
> > optimal for
> > the type of query you're doing.
>
> While this explanation may help for Beni's issue, my documents _are_
> fairly small with a minimum of duplicate elements, yet my queries are
> taking 3 -4 *minutes*. I finally broke down and tried this
> programmatically instead of from command line. The following program
> runs the query in about 3:30. However, thinking that the first query
> might be untypically slow, I tried again and had it run a few slightly
> different queries in one run, and they all take that long. Does anyone
> know where I can start to look for possible causes of this, and ways to
> improve?
>
> package foo;
>
> import org.dom4j.io.*;
> import org.xmldb.api.*;
> import org.xmldb.api.base.*;
> import org.xmldb.api.modules.*;
>
> public class QueryRunner {
> private static DOMReader xmlReader = new DOMReader();
> private static Database database = null;
>
> public static void main(String[] args) {
> String xpath = "/candidate[biographic_data/id='ANON2021']";
> org.w3c.dom.Node node;
> org.dom4j.Node newnode;
> try {
> Class xindiceDriver =
> Class.forName("org.apache.xindice.client.xmldb.DatabaseImpl");
> database = (Database) xindiceDriver.newInstance();
> DatabaseManager.registerDatabase(database);
> Collection col =
> DatabaseManager.getCollection("xmldb:xindice:///db/resumes");
> XPathQueryService service = (XPathQueryService)
> col.getService("XPathQueryService", "1.0");
> System.out.println(String.valueOf(new java.util.Date()));
> ResourceSet rs = service.query(xpath);
> System.out.println(String.valueOf(new java.util.Date()));
> ResourceIterator ri = rs.getIterator();
> while (ri.hasMoreResources()) {
> node = ((XMLResource) ri.nextResource()).getContentAsDOM();
> System.out.println(xmlReader.read((org.w3c.dom.Document)
> node).asXML());
> }
> }
> catch (Exception e) { e.printStackTrace(); }
> }
> }
--
Mark J Stang
System Architect
Cybershop Systems
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
I don't know the answers to these questions, as I'm still using 1.0. In
particular, the CVS version has some considerable differences with 1.0 --
it's always embedded, not a separate server, may not use CORBA (but rather
XML-RPC), so check it out carefully.
While you're thinking about it, you could convert the plain text to xml
files stored in a single directory. Then, when you've decided which version
of xindice to use, you could import the directory into a collection.
Alternatively, you could take your existing database, extract each resource
from the collection in question and stick it in a new collection with an
index already established. You could do this with the commandline tools or
a little java program (maybe even easier).
Jeff
----- Original Message -----
From: "Matthew Van Horn" <ho...@accesstech.com>
To: <xi...@xml.apache.org>
Sent: Monday, November 11, 2002 8:54 PM
Subject: Re: Performance question (Am I doing something wrong?)
> Well it sounds worth a try - I just wish my program for populating the
> db didn't take so long. It about an hour and a half to read a 8000
> plain text documents and convert them to XML for storage. Should I
> upgrade Xindice to the CVS version to avoid this bug? How does it
> compare to 1.0?
>
>
> On Tuesday, November 12, 2002, at 01:17 PM, Jeff Greif wrote:
>
> > Well, I'll try once more. It has been observed in
> > http://marc.theaimsgroup.com/?l=xindice-users&m=102155822118842&w=2
> > that owing to a bug (?) in xindice 1.0, under some or all
> > circumstances,
> > creating the index after adding documents does not work right. If you
> > observed no difference in query time with and without the index, you
> > probably need to delete the collection, create the index, and then add
> > the
> > documents again. In the cited message, query times dropped from 3
> > min. to
> > 30ms for a collection with 100,000 documents (or 120ms for 500,000
> > documents).
> >
> > Jeff
> > ----- Original Message -----
> > From: "Matthew Van Horn" <ho...@accesstech.com>
> > To: <xi...@xml.apache.org>
> > Sent: Monday, November 11, 2002 7:08 PM
> > Subject: Re: Performance question (Am I doing something wrong?)
> >
> >
> >> On Tuesday, November 12, 2002, at 11:47 AM, Jeff Greif wrote:
> >>
> >> While this explanation may help for Beni's issue, my documents _are_
> >> fairly small with a minimum of duplicate elements, yet my queries are
> >> taking 3 -4 *minutes*. I finally broke down and tried this
> >> programmatically instead of from command line. The following program
> >> runs the query in about 3:30. However, thinking that the first query
> >> might be untypically slow, I tried again and had it run a few slightly
> >> different queries in one run, and they all take that long. Does
> >> anyone
> >> know where I can start to look for possible causes of this, and ways
> >> to
> >> improve?
> >
>
>
Re: Performance question (Am I doing something wrong?)
Posted by Matthew Van Horn <ho...@accesstech.com>.
Well it sounds worth a try - I just wish my program for populating the
db didn't take so long. It about an hour and a half to read a 8000
plain text documents and convert them to XML for storage. Should I
upgrade Xindice to the CVS version to avoid this bug? How does it
compare to 1.0?
On Tuesday, November 12, 2002, at 01:17 PM, Jeff Greif wrote:
> Well, I'll try once more. It has been observed in
> http://marc.theaimsgroup.com/?l=xindice-users&m=102155822118842&w=2
> that owing to a bug (?) in xindice 1.0, under some or all
> circumstances,
> creating the index after adding documents does not work right. If you
> observed no difference in query time with and without the index, you
> probably need to delete the collection, create the index, and then add
> the
> documents again. In the cited message, query times dropped from 3
> min. to
> 30ms for a collection with 100,000 documents (or 120ms for 500,000
> documents).
>
> Jeff
> ----- Original Message -----
> From: "Matthew Van Horn" <ho...@accesstech.com>
> To: <xi...@xml.apache.org>
> Sent: Monday, November 11, 2002 7:08 PM
> Subject: Re: Performance question (Am I doing something wrong?)
>
>
>> On Tuesday, November 12, 2002, at 11:47 AM, Jeff Greif wrote:
>>
>> While this explanation may help for Beni's issue, my documents _are_
>> fairly small with a minimum of duplicate elements, yet my queries are
>> taking 3 -4 *minutes*. I finally broke down and tried this
>> programmatically instead of from command line. The following program
>> runs the query in about 3:30. However, thinking that the first query
>> might be untypically slow, I tried again and had it run a few slightly
>> different queries in one run, and they all take that long. Does
>> anyone
>> know where I can start to look for possible causes of this, and ways
>> to
>> improve?
>
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
Well, I'll try once more. It has been observed in
http://marc.theaimsgroup.com/?l=xindice-users&m=102155822118842&w=2
that owing to a bug (?) in xindice 1.0, under some or all circumstances,
creating the index after adding documents does not work right. If you
observed no difference in query time with and without the index, you
probably need to delete the collection, create the index, and then add the
documents again. In the cited message, query times dropped from 3 min. to
30ms for a collection with 100,000 documents (or 120ms for 500,000
documents).
Jeff
----- Original Message -----
From: "Matthew Van Horn" <ho...@accesstech.com>
To: <xi...@xml.apache.org>
Sent: Monday, November 11, 2002 7:08 PM
Subject: Re: Performance question (Am I doing something wrong?)
> On Tuesday, November 12, 2002, at 11:47 AM, Jeff Greif wrote:
>
> While this explanation may help for Beni's issue, my documents _are_
> fairly small with a minimum of duplicate elements, yet my queries are
> taking 3 -4 *minutes*. I finally broke down and tried this
> programmatically instead of from command line. The following program
> runs the query in about 3:30. However, thinking that the first query
> might be untypically slow, I tried again and had it run a few slightly
> different queries in one run, and they all take that long. Does anyone
> know where I can start to look for possible causes of this, and ways to
> improve?
Re: Performance question (Am I doing something wrong?)
Posted by Matthew Van Horn <ho...@accesstech.com>.
On Tuesday, November 12, 2002, at 11:47 AM, Jeff Greif wrote:
> I think I see the problem. I believe (if I've read the source code
> correctly) that in xindice, your index on <w> is essentially a map from
> values of <w> to document keys.
> ...snippage...
> Clearly, if I'm not confused, xindice is optimized for small documents
> with
> not much repeating structure, and its indexing mechanism is not
> optimal for
> the type of query you're doing.
While this explanation may help for Beni's issue, my documents _are_
fairly small with a minimum of duplicate elements, yet my queries are
taking 3 -4 *minutes*. I finally broke down and tried this
programmatically instead of from command line. The following program
runs the query in about 3:30. However, thinking that the first query
might be untypically slow, I tried again and had it run a few slightly
different queries in one run, and they all take that long. Does anyone
know where I can start to look for possible causes of this, and ways to
improve?
package foo;
import org.dom4j.io.*;
import org.xmldb.api.*;
import org.xmldb.api.base.*;
import org.xmldb.api.modules.*;
public class QueryRunner {
private static DOMReader xmlReader = new DOMReader();
private static Database database = null;
public static void main(String[] args) {
String xpath = "/candidate[biographic_data/id='ANON2021']";
org.w3c.dom.Node node;
org.dom4j.Node newnode;
try {
Class xindiceDriver =
Class.forName("org.apache.xindice.client.xmldb.DatabaseImpl");
database = (Database) xindiceDriver.newInstance();
DatabaseManager.registerDatabase(database);
Collection col =
DatabaseManager.getCollection("xmldb:xindice:///db/resumes");
XPathQueryService service = (XPathQueryService)
col.getService("XPathQueryService", "1.0");
System.out.println(String.valueOf(new java.util.Date()));
ResourceSet rs = service.query(xpath);
System.out.println(String.valueOf(new java.util.Date()));
ResourceIterator ri = rs.getIterator();
while (ri.hasMoreResources()) {
node = ((XMLResource) ri.nextResource()).getContentAsDOM();
System.out.println(xmlReader.read((org.w3c.dom.Document)
node).asXML());
}
}
catch (Exception e) { e.printStackTrace(); }
}
}
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
I think I see the problem. I believe (if I've read the source code
correctly) that in xindice, your index on <w> is essentially a map from
values of <w> to document keys. When the documents mapping matching the
constraint on the index are found, an XPath selection of the desired nodes
in the document is then carried out. Since your 3 documents in the original
query each had thousands of sentences, and each of the sentences in the
subset of the documents selected by the index had to then be checked to see
if it contained the word in question, this could be very expensive.
Clearly, if I'm not confused, xindice is optimized for small documents with
not much repeating structure, and its indexing mechanism is not optimal for
the type of query you're doing.
A possible organization of your data in xindice that would make the query go
*much* faster is to make each sentence a document, adding to the <s>
elements the document key and the number of the sentence in the document
(perhaps the sentence id already represents this information?). Instead of
having a collection of documents you would have a collection of sentences.
With this organization you can rapidly find sentences containing a given
word, and you can easily reassemble a document.
Presumably, fancier indexing structures could be devised for larger
documents similar to the kind you've tried here, where the index maps values
to document + unique xpath. Here's a possibility I'm inventing on the fly.
Suppose all elements were internally assigned unique identifiers within a
document. You could then use the index to map the value to pairs like
(doc-id,
/doc[@uid='doc1']/body[@uid='body1']/s[@uid='s63']/w[@uid='w7']) and have a
byte-valued (or node-valued) index on uid. The following procedure would
then rapidly find the right nodes. The xpath in the tuples retrieved from
the index would be matched against the node selection part of the query
xpath, e.g. /doc/body/s. This would determine that we're supposed to find
the node with uid='s63', and we could go right to the node in the dom or
byte in a serialized representation. I don't know if anyone has plans for
such indexing structures in xindice -- they look pretty expensive to
maintain when you allow update, and they would take up a non-trivial amount
of storage.
Jeff
----- Original Message -----
From: "Beni Ruef" <br...@es.unizh.ch>
To: <xi...@xml.apache.org>
Cc: "Hans Martin Lehmann" <hm...@es.unizh.ch>; "Sebastian Hoffmann"
<se...@es.unizh.ch>
Sent: Monday, November 11, 2002 1:44 PM
Subject: Re: Performance question (Am I doing something wrong?)
> On Sunday, November 10, 2002, at 08:45, Mark J. Stang wrote:
>
> > Not wrong. If that is what your tests show, then you are not doing
> > anything wrong. Not if your test is "how long will a command-line
> > query take?"
>
> Got your point, excuse my stupidity ;-)
>
> > Try adding in a thousand such documents and
> > index them. My guess is that the query time will increase but
> > it will be hard to measure the increase.
>
> That's correct but there's a big but: searching for rare words (as my
> example was) is indeed rather independent of the corpus size but
> searching for more common words is another story (the following numbers
> are with a 30 MB corpus):
> Searching for a word occuring only twice takes ca. 3 seconds, searching
> for a word occuring 13 times takes 70 seconds, and a word occuring 1300
> times takes even more than 6 minutes!
>
> This is a typical characteristic of linguistic corpora: although the
> number of words is virtually unlimited the number of unique word forms
> is quite restricted: one of my corpora has 106'000 words but only 19'000
> different word forms. It's even worse with word categories or POS tags
> (the 'pos' attributes in my example) as there are just 50 - 100 of
> them. Technically speaking, the values used as keys in the index are
> far from unique -- I don't know how Xindice can handle this problem.
>
> > The only accurate way to measure the time is to build a small
> > program and test the actual call to the database. Try using
> > Example1.java, that one has always worked well for me.
>
> I did this and it takes two seconds (measured between getCollection()
> and getIterator()) for my "rare word" query.
>
> > I had a problem with Mac OS X with every query taking
> > forever. Turned out that the database initialization was being
> > done every time and it took forever.
>
> But why, isn't the Java code identical? (Disclaimer: I know nothing
> about Java ;-)
>
> > The same query was faster on
> > Windows and Linux.
>
> I did my benchmarks on a slow Linux machine (AMD-K6 350 MHz) and they
> take only twice as long as on my (fast :-) iBook.
>
> > I ended up caching my collections.
>
> How do you do this?
>
> > This is from an e-mail dated September 5, 2001.
> >
> > Kimbro Staken wrote:
> >
> >> computer: 750MHZ P3 256MB RAM Laptop running Mandrake Linux 8
> >> jdk: Sun 1.3.0_04
> >> Dataset size: 149,025 documents 601MB
> >> Insertion time (no indexes): 1 hour 45 minutes which is roughly 1,424
> >> docs
> >> per minute or 24 per second.
> >> Collection size: 657MB
> >> Document retrieval: 2 seconds (including VM startup which is most of
> >> the
> >> time)
> >> Full collection scan query /disc[id = '11041c03']: 12 minutes
> >> Index creation: 13.5 minutes
> >> Index based query /disc[id = '11041c03']: 2.12 seconds (including VM
> >> startup which is most of that time)
>
> I'd really like such a fast response :-) but these are unique keys...
>
> Cheers
> -Beni
>
>
Re: Performance question (Am I doing something wrong?)
Posted by Beni Ruef <br...@es.unizh.ch>.
On Sunday, November 10, 2002, at 08:45, Mark J. Stang wrote:
> Not wrong. If that is what your tests show, then you are not doing
> anything wrong. Not if your test is "how long will a command-line
> query take?"
Got your point, excuse my stupidity ;-)
> Try adding in a thousand such documents and
> index them. My guess is that the query time will increase but
> it will be hard to measure the increase.
That's correct but there's a big but: searching for rare words (as my
example was) is indeed rather independent of the corpus size but
searching for more common words is another story (the following numbers
are with a 30 MB corpus):
Searching for a word occuring only twice takes ca. 3 seconds, searching
for a word occuring 13 times takes 70 seconds, and a word occuring 1300
times takes even more than 6 minutes!
This is a typical characteristic of linguistic corpora: although the
number of words is virtually unlimited the number of unique word forms
is quite restricted: one of my corpora has 106'000 words but only 19'000
different word forms. It's even worse with word categories or POS tags
(the 'pos' attributes in my example) as there are just 50 - 100 of
them. Technically speaking, the values used as keys in the index are
far from unique -- I don't know how Xindice can handle this problem.
> The only accurate way to measure the time is to build a small
> program and test the actual call to the database. Try using
> Example1.java, that one has always worked well for me.
I did this and it takes two seconds (measured between getCollection()
and getIterator()) for my "rare word" query.
> I had a problem with Mac OS X with every query taking
> forever. Turned out that the database initialization was being
> done every time and it took forever.
But why, isn't the Java code identical? (Disclaimer: I know nothing
about Java ;-)
> The same query was faster on
> Windows and Linux.
I did my benchmarks on a slow Linux machine (AMD-K6 350 MHz) and they
take only twice as long as on my (fast :-) iBook.
> I ended up caching my collections.
How do you do this?
> This is from an e-mail dated September 5, 2001.
>
> Kimbro Staken wrote:
>
>> computer: 750MHZ P3 256MB RAM Laptop running Mandrake Linux 8
>> jdk: Sun 1.3.0_04
>> Dataset size: 149,025 documents 601MB
>> Insertion time (no indexes): 1 hour 45 minutes which is roughly 1,424
>> docs
>> per minute or 24 per second.
>> Collection size: 657MB
>> Document retrieval: 2 seconds (including VM startup which is most of
>> the
>> time)
>> Full collection scan query /disc[id = '11041c03']: 12 minutes
>> Index creation: 13.5 minutes
>> Index based query /disc[id = '11041c03']: 2.12 seconds (including VM
>> startup which is most of that time)
I'd really like such a fast response :-) but these are unique keys...
Cheers
-Beni
Re: Performance question (Am I doing something wrong?)
Posted by "Mark J. Stang" <ma...@earthlink.net>.
Not wrong. If that is what your tests show, then you are not doing
anything wrong. Not if your test is "how long will a command-line
query take?" The question is, is that your entire application and
use of Xindice. Try adding in a thousand such documents and
index them. My guess is that the query time will increase but
it will be hard to measure the increase.
Xindice uses a binary tree look up. Even if it didn't, on your
machine you could open the file, read the entire document
into memory and use a java sub-string search in less time.
The only accurate way to measure the time is to build a small
program and test the actual call to the database. Try using
Example1.java, that one has always worked well for me.
I had a problem with Mac OS X with every query taking
forever. Turned out that the database initialization was being
done every time and it took forever. Ran the same program
on a single processor Mac and a dual-processor Mac. One
was fast the other slow. The same query was faster on
Windows and Linux. I ended up caching my collections.
This is from an e-mail dated September 5, 2001.
Kimbro Staken wrote:
> As I've been working out some issues with the CORBA system I've been
> working on getting larger document sets into the server. My largest set
> right now is 149,025 documents in a single collection. The server can
> easily handle more documents this is just the largest dataset I have
> available right now. Here are some stats to give us a better idea where we
> stand. These are run against the current CVS version with one exception. I
> used OpenORB for the server ORB instead of JacORB. JacORB was still used
> for the client. It's likely we'll need to switch to OpenORB overall as
> even the latest JacORB leaks memory on the server.
>
> computer: 750MHZ P3 256MB RAM Laptop running Mandrake Linux 8
> jdk: Sun 1.3.0_04
> Dataset size: 149,025 documents 601MB
> Insertion time (no indexes): 1 hour 45 minutes which is roughly 1,424 docs
> per minute or 24 per second.
> Collection size: 657MB
> Document retrieval: 2 seconds (including VM startup which is most of the
> time)
> Full collection scan query /disc[id = '11041c03']: 12 minutes
> Index creation: 13.5 minutes
> Index based query /disc[id = '11041c03']: 2.12 seconds (including VM
> startup which is most of that time)
> Index size 164MB
>
> The data set consists of documents similar to the following.
>
> <?xml version="1.0"?>
> <disc>
> <id>11041c03</id>
> <length>1054</length>
> <title>Orchestral Manoeuvres In The Dark / The OMD Remixes (Single)</title>
> <genre>cddb/misc</genre>
> <track index="1" offset="150">Enola Gay (OMD vs Sash! Radio Edit)</track>
> <track index="2" offset="18790"> (2)Souvenir (Moby Remix)</track>
> <track index="3" offset="39790"> (3)Electricity (The Micronauts
> Remix)</track>
> </disc>
>
> Kimbro Staken
> The dbXML Group L.L.C. - http://www.dbxmlgroup.com/
> Embedded XML Database Software and Services
>
> _______________________________________________
> dbXML-Core-Devel mailing list
> dbXML-Core-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dbxml-core-devel
HTH,
Mark
Beni Ruef wrote:
> I just installed Xindice 1.0 on my iBook under Mac OS X (with enough
> RAM, i.e. a reasonably fast machine).
>
> The (TEI encoded) texts I'm interested in look like this:
>
> .
> .
> <s n="id-1.1"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
> pos="AT1">a</w><w pos="NN1">sentence</w><c pos="YSTP">.</c></s>
> <s n="id-1.2"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
> pos="DD1">another</w><w pos="PN1">one</w><c pos="YSTP">.</c></s>
> </body></text>
> </TEI.2>
>
> and the simplest queries like this:
>
> xindice xpath -c /db/myCollection -q
> '/TEI.2/text/body/s[w="another"]'
>
> With a test corpus containing three documents and a total of ca. 700
> KBytes the above query takes ca. 9 seconds...
> More surprisingly, even a simple retrieval (xindice rd) of a 200K
> document needs 5 seconds!
>
> After having run
>
> xindiceadmin ai -c /db/myCollection -p w -n wordform
>
> things improve slightly as the same query takes now 4.5 seconds, a query
> with a non-existing word form "only" 2.5 seconds. BTW, there seems to
> be an overhead of ca. 2 seconds as any operation takes at least 2
> seconds...
>
> Obviously, this is still way too slow to be usable as I'm planning to
> work with corpora containing some 100 million words...
>
> So what am I doing (terribly ;-) wrong and what can be improved?! What
> about this overhead and what about the indexer switches like pagesize?
>
> Thanks in advance, Cheers
> -Beni
--
Mark J Stang
System Architect
Cybershop Systems
Re: Performance question (Am I doing something wrong?)
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
Each invocation of the command line tool causes a process to be started,
maybe some configuration files to be read, and a CORBA connection made.
Probably there is one second or more associated with that.
You might try running under more realistic conditions for a production app
by taking one of the sample apps from the developer's guide and modifying it
to run a set of queries (possibly read from a file or the command line) in a
loop, perhaps with a varying number of passes through the list, possibly
randomly selecting a query on each attempt. Try to make the set of queries
large enough so all the results won't be cached when you repeat, or run the
test twice, once with caching off if this can be done, to see the
difference.
I would have expected a much larger improvement using the index. It's
conceivable that an index on 's' might help also.
It would also be good to test with more documents to see how things scale,
since you can't tell this from what you have in the collection now.
Just out of curiosity, what part-of-speech tagger are you using?
Jeff
----- Original Message -----
From: "Beni Ruef" <br...@es.unizh.ch>
To: <xi...@xml.apache.org>
Sent: Sunday, November 10, 2002 6:50 AM
Subject: Performance question (Am I doing something wrong?)
> I just installed Xindice 1.0 on my iBook under Mac OS X (with enough
> RAM, i.e. a reasonably fast machine).
>
> The (TEI encoded) texts I'm interested in look like this:
>
> .
> .
> <s n="id-1.1"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
> pos="AT1">a</w><w pos="NN1">sentence</w><c pos="YSTP">.</c></s>
> <s n="id-1.2"><w pos="DD1">This</w> <w pos="VBZ">is</w> <w
> pos="DD1">another</w><w pos="PN1">one</w><c pos="YSTP">.</c></s>
> </body></text>
> </TEI.2>
>
> and the simplest queries like this:
>
> xindice xpath -c /db/myCollection -q
> '/TEI.2/text/body/s[w="another"]'
>
> With a test corpus containing three documents and a total of ca. 700
> KBytes the above query takes ca. 9 seconds...
> More surprisingly, even a simple retrieval (xindice rd) of a 200K
> document needs 5 seconds!
>
> After having run
>
> xindiceadmin ai -c /db/myCollection -p w -n wordform
>
> things improve slightly as the same query takes now 4.5 seconds, a query
> with a non-existing word form "only" 2.5 seconds. BTW, there seems to
> be an overhead of ca. 2 seconds as any operation takes at least 2
> seconds...
>
> Obviously, this is still way too slow to be usable as I'm planning to
> work with corpora containing some 100 million words...
>
> So what am I doing (terribly ;-) wrong and what can be improved?! What
> about this overhead and what about the indexer switches like pagesize?
>
> Thanks in advance, Cheers
> -Beni
>
>