You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2007/06/22 22:18:10 UTC
threads & benchmark contrib
Hi,
I'm trying to test LUCENE-843 (IndexWriter speedups) on Wikipedia
using the the benchmark contrib framework plus the patch from
LUCENE-848.
I downloaded an older wikipedia export (the "latest" doesn't seem to
exist) and got it un-tar'd. The test I'd like to run is to use 4
threads to index all (exhaust) documents. I'm using the alg below.
One problem I hit is the DirDocMaker uses a SimpleDateFormat instance
for parsing the dates at the top of each file, but, this is not
threadsafe and so I hit exceptions from there. I think we just need
to make that instance thread local I think (I will open issue).
The question I have is: is this alg going to do what I want? I'd like
each doc in Wikipedia to be indexed only once, with 4 threads running.
I *think* but I'm not sure that the alg below actually indexes the
Wikipedia content 4 times over instead?
Here's the alg:
max.field.length=2147483647
compound=false
analyzer=org.apache.lucene.analysis.SimpleAnalyzer
directory=FSDirectory
# ram.flush.mb=32
max.buffered=10000
doc.stored=true
doc.tokenized=true
doc.term.vector=true
doc.add.log.step=500
docs.dir=enwiki
doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker
# task at this depth or less would print when they start
task.max.depth.log=1
doc.maker.forever=false
# -------------------------------------------------------------------------------------
ResetSystemErase
CreateIndex
{[AddDoc(4000)]: 4} : *
CloseIndex
RepSumByPref AddDoc
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: threads & benchmark contrib
Posted by Michael McCandless <lu...@mikemccandless.com>.
"Doron Cohen" <DO...@il.ibm.com> wrote:
> Mike, I didn't anticipate this use case and I think it
> would not work correctly. I'll look into this.
>
> Anyhow, I think it would not work as you expect.
>
> It seems what you want is to have 4 threads, adding docs in
> parallel, until the doc maker is exhausted.
>
> But this line:
> {[AddDoc(4000)]: 4} : *
>
> Reads as -
> Repeatedly until exhausted:
> Create & Start 4 threads (in parallel),
> each adding 1 doc of size 4000;
> Wait for them 4 threads to complete.
>
> Now, this is not what you are after, is it? I think
> you would like just 4 threads to do all the work.
Ahh, right. I had the two tasks "inverted".
> It seems what you are really after is this:
> [ { AddDoc } : * ] : 4
>
> This reads as:
> Create 4 threads, each adding docs until exhaustion.
>
> Since there is a single system-benchmark-wide doc-maker, all 4
> threads use it, and when it is exhausted, all 4 will be done.
>
> I tried this way and it works as I expected it to (except
> for that DateFormat bug, see below). Can you try like this
> and let me know if it works for you.
Super, I will test this one! Thanks.
> I think your variation of this exposes a bug in the
> benchmark - it will just loop forever because the parallel
> sequence would mask the exhaustion from the outer sequential
> sequence. I opened LUCENE-941 for this, and looking into it
OK, thanks. The benchmark framework is very rich/expressive!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: threads & benchmark contrib
Posted by Doron Cohen <DO...@il.ibm.com>.
Mike, I didn't anticipate this use case and I think it
would not work correctly. I'll look into this.
Anyhow, I think it would not work as you expect.
It seems what you want is to have 4 threads, adding docs in
parallel, until the doc maker is exhausted.
But this line:
{[AddDoc(4000)]: 4} : *
Reads as -
Repeatedly until exhausted:
Create & Start 4 threads (in parallel),
each adding 1 doc of size 4000;
Wait for them 4 threads to complete.
Now, this is not what you are after, is it? I think
you would like just 4 threads to do all the work.
It seems what you are really after is this:
[ { AddDoc } : * ] : 4
This reads as:
Create 4 threads, each adding docs until exhaustion.
Since there is a single system-benchmark-wide doc-maker, all 4
threads use it, and when it is exhausted, all 4 will be done.
I tried this way and it works as I expected it to (except
for that DateFormat bug, see below). Can you try like this
and let me know if it works for you.
I think your variation of this exposes a bug in the
benchmark - it will just loop forever because the parallel
sequence would mask the exhaustion from the outer sequential
sequence. I opened LUCENE-941 for this, and looking into it
Doron
"Michael McCandless" <lu...@mikemccandless.com> wrote on 22/06/2007
13:18:10:
>
> Hi,
>
> I'm trying to test LUCENE-843 (IndexWriter speedups) on Wikipedia
> using the the benchmark contrib framework plus the patch from
> LUCENE-848.
>
> I downloaded an older wikipedia export (the "latest" doesn't seem to
> exist) and got it un-tar'd. The test I'd like to run is to use 4
> threads to index all (exhaust) documents. I'm using the alg below.
>
> One problem I hit is the DirDocMaker uses a SimpleDateFormat instance
> for parsing the dates at the top of each file, but, this is not
> threadsafe and so I hit exceptions from there. I think we just need
> to make that instance thread local I think (I will open issue).
Yes, tha's a bug... It is also in some already committed parts
of the benchmark. I opened LUCENE-940 for this.
>
> The question I have is: is this alg going to do what I want? I'd like
> each doc in Wikipedia to be indexed only once, with 4 threads running.
> I *think* but I'm not sure that the alg below actually indexes the
> Wikipedia content 4 times over instead?
>
> Here's the alg:
>
> max.field.length=2147483647
> compound=false
>
> analyzer=org.apache.lucene.analysis.SimpleAnalyzer
> directory=FSDirectory
> # ram.flush.mb=32
> max.buffered=10000
> doc.stored=true
> doc.tokenized=true
> doc.term.vector=true
> doc.add.log.step=500
>
> docs.dir=enwiki
>
> doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker
>
> # task at this depth or less would print when they start
> task.max.depth.log=1
> doc.maker.forever=false
>
> #
>
-------------------------------------------------------------------------------------
>
> ResetSystemErase
> CreateIndex
> {[AddDoc(4000)]: 4} : *
> CloseIndex
>
> RepSumByPref AddDoc
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org