You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2007/10/17 16:42:01 UTC
contrib/benchmark Parallel tasks ?
Hi,
I am using the contrib/benchmarker to do some performance tests. I
have a 4 core machine, so I would like to test using 4 threads for
indexing. From the docs at http://lucene.zones.apache.org:8080/
hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/benchmark/byTask/
package-summary.html
I then want to modify the conf/standard.alg file to be just indexing,
and to do that in a multithreaded way.
I believe this means, I should change:
{ "MAddDocs" AddDoc } : 20000
to be:
[ "MAddDocs" AddDoc ] : 20000 : 4
right?
I guess I am confused a bit by the rate factor documentation in the
docs given by the URL above, for instance, it says:
"Example - [ AddDoc ] : 400 : 3 - would do 400 addDoc in parallel,
starting up to 3 threads per second. "
but, I think I want instead: start up 4 threads, and then have each
split up the indexing of 20,000 documents. Thus, each thread would
index 5K docs.
Am I expressing this correctly? Something doesn't quite feel right
about this.
Thanks,
Grant
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Boot Camp Training:
ApacheCon Atlanta, Nov. 12, 2007. Sign up now! http://
www.apachecon.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: contrib/benchmark Parallel tasks ?
Posted by Doron Cohen <DO...@il.ibm.com>.
Hi Grant,
Grant Ingersoll wrote:
> I think the answer is:
> [{ "MAddDocs" AddDoc } : 5000] : 4
>
> Is this the functional equivalent of doing:
> { "MAddDocs" AddDoc } : 20000
>
> in parallel?
Yes, this is correct, it reads as "create 4 threads, each
adding 5000 docs to the index, and start/run the 4 threads
in parallel.
Note that a single doc maker is serving all the threads.
So another possible way to run this is:
[ { AddDoc } : * ] : 4
This reads as: "create 4 threads, each adding docs until
the (single) doc maker is exhausted, and start/run the 4
threads in parallel. This was also discussed in
http://www.mail-archive.com/java-dev@lucene.apache.org/msg10831.html
("threads & benchmark contrib") and so these test cases are also
relevant: TestPerfTasksLogic's testParallelDocMaker()
and testParallelExhausted().
Regards,
Doron
>
> Thanks,
> Grant
>
> On Oct 17, 2007, at 10:42 AM, Grant Ingersoll wrote:
>
> > Hi,
> >
> > I am using the contrib/benchmarker to do some performance tests. I
> > have a 4 core machine, so I would like to test using 4 threads for
> > indexing. From the docs at http://lucene.zones.apache.org:8080/
> > hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/benchmark/
> > byTask/package-summary.html
> >
> > I then want to modify the conf/standard.alg file to be just
> > indexing, and to do that in a multithreaded way.
> >
> > I believe this means, I should change:
> > { "MAddDocs" AddDoc } : 20000
> >
> > to be:
> > [ "MAddDocs" AddDoc ] : 20000 : 4
> >
> > right?
> >
> > I guess I am confused a bit by the rate factor documentation in the
> > docs given by the URL above, for instance, it says:
> > "Example - [ AddDoc ] : 400 : 3 - would do 400 addDoc in parallel,
> > starting up to 3 threads per second. "
> >
> > but, I think I want instead: start up 4 threads, and then have each
> > split up the indexing of 20,000 documents. Thus, each thread would
> > index 5K docs.
> >
> > Am I expressing this correctly? Something doesn't quite feel right
> > about this.
> >
> > Thanks,
> > Grant
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: contrib/benchmark Parallel tasks ?
Posted by Grant Ingersoll <gs...@apache.org>.
I think the answer is:
[{ "MAddDocs" AddDoc } : 5000] : 4
Is this the functional equivalent of doing:
{ "MAddDocs" AddDoc } : 20000
in parallel?
Thanks,
Grant
On Oct 17, 2007, at 10:42 AM, Grant Ingersoll wrote:
> Hi,
>
> I am using the contrib/benchmarker to do some performance tests. I
> have a 4 core machine, so I would like to test using 4 threads for
> indexing. From the docs at http://lucene.zones.apache.org:8080/
> hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/benchmark/
> byTask/package-summary.html
>
> I then want to modify the conf/standard.alg file to be just
> indexing, and to do that in a multithreaded way.
>
> I believe this means, I should change:
> { "MAddDocs" AddDoc } : 20000
>
> to be:
> [ "MAddDocs" AddDoc ] : 20000 : 4
>
> right?
>
> I guess I am confused a bit by the rate factor documentation in the
> docs given by the URL above, for instance, it says:
> "Example - [ AddDoc ] : 400 : 3 - would do 400 addDoc in parallel,
> starting up to 3 threads per second. "
>
> but, I think I want instead: start up 4 threads, and then have each
> split up the indexing of 20,000 documents. Thus, each thread would
> index 5K docs.
>
> Am I expressing this correctly? Something doesn't quite feel right
> about this.
>
> Thanks,
> Grant
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Boot Camp Training:
> ApacheCon Atlanta, Nov. 12, 2007. Sign up now! http://
> www.apachecon.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
Lucene Boot Camp Training:
ApacheCon Atlanta, Nov. 12, 2007. Sign up now! http://
www.apachecon.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org