You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2024/06/28 15:33:03 UTC

Solr "benchmark", wikipedia

I was thinking of using Solr's "benchmark" module/thing to benchmark
parallel segment search (coming to Solr 9.7 but needs more love).  I
don't notice any substantial data to query for in this module,
however.  Has anyone considered adding wikipedia, like how Lucene's
"luceneutil" does?  Or something else?  Is this a bad idea for this
benchmark module or should I be looking elsewhere like Searchscale's
solr-bench[1]?

https://github.com/searchscale/solr-bench

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org


Re: Solr "benchmark", wikipedia

Posted by Mark Miller <ma...@gmail.com>.
The benchmark module is focused on micro-benchmarks - benchmarking specific
features or specific code. These kinds of benchmarks are notoriously
difficult to get right in Java, and the benchmark module is built on JMH,
in order to make it much easier to get things right.

The module has a built-in random data generator so that you can easily
target whatever kind of data you would like - not just text fields but
faceting fields or whatever. For text fields, you would generally use
something like a zipf distribution from the English word dictionary.

Re: Solr "benchmark", wikipedia

Posted by Mark Miller <ma...@gmail.com>.
The benchmark module is focused on micro-benchmarks - benchmarking specific
features or specific code. These kinds of benchmarks are notoriously
difficult to get right in Java, and the benchmark module is built on JMH,
in order to make it much easier to get things right.

The module has a built-in random data generator so that you can easily
target whatever kind of data you would like - not just text fields but
faceting fields or whatever. For text fields, you would generally use
something like a zipf distribution from the English word dictionary.

Re: Solr "benchmark", wikipedia

Posted by Jason Gerlowski <ge...@gmail.com>.
I've found the "benchmark" module easier to extend and write custom
benchmarks with.  And from what I recall it was easier to attach
profilers, get flamegraphs, etc - which seems useful if there are
questions around parallel-segment search.

But it depends on exactly what you want.  I've found solr-bench to be
useful in other scenarios as well.

On Fri, Jun 28, 2024 at 4:03 PM Ishan Chattopadhyaya
<ic...@gmail.com> wrote:
>
> Might need to install the prerequisites on Linux as follows:
>
> apt install wget unzip zip ant ivy lsof git netcat make openjdk-11-jdk
> maven jq
>
> Something similar for MacOS can be installed using homebrew etc. The jq and
> maven are important.
>
> On Sat, 29 Jun, 2024, 1:04 am Ishan Chattopadhyaya, <
> ichattopadhyaya@gmail.com> wrote:
>
> > Off the top of my head:
> >
> > git clone https://GitHub.com/fullstorydev/solr-bench
> > cd solr-bench
> > git checkout ishan/local-mode-fixes
> >
> > mvn clean compile assembly:single
> > ./cleanup.sh && ./stress.sh -c <commit> suites/stress-facets-local.json
> >
> > (Replace commit with the commit id hash of the latest tip of branch_9x)
> >
> > On Sat, 29 Jun, 2024, 12:32 am Ishan Chattopadhyaya, <
> > ichattopadhyaya@gmail.com> wrote:
> >
> >> Hi David,
> >> I'll publish benchmarks and steps to reproduce tests we are doing using
> >> solr-bench. Let's collaborate a bit more to make sure we can extract
> >> maximum performance out of the feature. Right now, I'm not seeing good
> >> numbers and it is a bit worrisome.
> >> Thanks and regards,
> >> Ishan
> >>
> >> On Sat, 29 Jun, 2024, 12:14 am Gus Heck, <gu...@gmail.com> wrote:
> >>
> >>> To comment further, I think it would be good if we had query benchmarks
> >>> that tried to map directly to the queries benchmarked for lucene, against
> >>> the same data. This would give us a notion of which half of the equation
> >>> any slow down comes from (or speed up!)
> >>>
> >>> On Fri, Jun 28, 2024 at 2:42 PM Gus Heck <gu...@gmail.com> wrote:
> >>>
> >>> > Yes. In fact I have an example in JesterJ of indexing the luceneutil
> >>> > data... (but it's still in early stages, I think it's still against
> >>> > _default schema, perhaps... (need to look again, haven't had time to
> >>> work
> >>> > on it recently)
> >>> >
> >>> > https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs
> >>> >
> >>> > On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org>
> >>> wrote:
> >>> >
> >>> >> I was thinking of using Solr's "benchmark" module/thing to benchmark
> >>> >> parallel segment search (coming to Solr 9.7 but needs more love).  I
> >>> >> don't notice any substantial data to query for in this module,
> >>> >> however.  Has anyone considered adding wikipedia, like how Lucene's
> >>> >> "luceneutil" does?  Or something else?  Is this a bad idea for this
> >>> >> benchmark module or should I be looking elsewhere like Searchscale's
> >>> >> solr-bench[1]?
> >>> >>
> >>> >> https://github.com/searchscale/solr-bench
> >>> >>
> >>> >> ~ David Smiley
> >>> >> Apache Lucene/Solr Search Developer
> >>> >> http://www.linkedin.com/in/davidwsmiley
> >>> >>
> >>> >> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >>> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>> >>
> >>> >>
> >>> >
> >>> > --
> >>> > http://www.needhamsoftware.com (work)
> >>> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >>> >
> >>>
> >>>
> >>> --
> >>> http://www.needhamsoftware.com (work)
> >>> https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >>>
> >>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org


Re: Solr "benchmark", wikipedia

Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Might need to install the prerequisites on Linux as follows:

apt install wget unzip zip ant ivy lsof git netcat make openjdk-11-jdk
maven jq

Something similar for MacOS can be installed using homebrew etc. The jq and
maven are important.

On Sat, 29 Jun, 2024, 1:04 am Ishan Chattopadhyaya, <
ichattopadhyaya@gmail.com> wrote:

> Off the top of my head:
>
> git clone https://GitHub.com/fullstorydev/solr-bench
> cd solr-bench
> git checkout ishan/local-mode-fixes
>
> mvn clean compile assembly:single
> ./cleanup.sh && ./stress.sh -c <commit> suites/stress-facets-local.json
>
> (Replace commit with the commit id hash of the latest tip of branch_9x)
>
> On Sat, 29 Jun, 2024, 12:32 am Ishan Chattopadhyaya, <
> ichattopadhyaya@gmail.com> wrote:
>
>> Hi David,
>> I'll publish benchmarks and steps to reproduce tests we are doing using
>> solr-bench. Let's collaborate a bit more to make sure we can extract
>> maximum performance out of the feature. Right now, I'm not seeing good
>> numbers and it is a bit worrisome.
>> Thanks and regards,
>> Ishan
>>
>> On Sat, 29 Jun, 2024, 12:14 am Gus Heck, <gu...@gmail.com> wrote:
>>
>>> To comment further, I think it would be good if we had query benchmarks
>>> that tried to map directly to the queries benchmarked for lucene, against
>>> the same data. This would give us a notion of which half of the equation
>>> any slow down comes from (or speed up!)
>>>
>>> On Fri, Jun 28, 2024 at 2:42 PM Gus Heck <gu...@gmail.com> wrote:
>>>
>>> > Yes. In fact I have an example in JesterJ of indexing the luceneutil
>>> > data... (but it's still in early stages, I think it's still against
>>> > _default schema, perhaps... (need to look again, haven't had time to
>>> work
>>> > on it recently)
>>> >
>>> > https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs
>>> >
>>> > On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org>
>>> wrote:
>>> >
>>> >> I was thinking of using Solr's "benchmark" module/thing to benchmark
>>> >> parallel segment search (coming to Solr 9.7 but needs more love).  I
>>> >> don't notice any substantial data to query for in this module,
>>> >> however.  Has anyone considered adding wikipedia, like how Lucene's
>>> >> "luceneutil" does?  Or something else?  Is this a bad idea for this
>>> >> benchmark module or should I be looking elsewhere like Searchscale's
>>> >> solr-bench[1]?
>>> >>
>>> >> https://github.com/searchscale/solr-bench
>>> >>
>>> >> ~ David Smiley
>>> >> Apache Lucene/Solr Search Developer
>>> >> http://www.linkedin.com/in/davidwsmiley
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> >> For additional commands, e-mail: dev-help@solr.apache.org
>>> >>
>>> >>
>>> >
>>> > --
>>> > http://www.needhamsoftware.com (work)
>>> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
>>> >
>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>>>
>>

Re: Solr "benchmark", wikipedia

Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Off the top of my head:

git clone https://GitHub.com/fullstorydev/solr-bench
cd solr-bench
git checkout ishan/local-mode-fixes

mvn clean compile assembly:single
./cleanup.sh && ./stress.sh -c <commit> suites/stress-facets-local.json

(Replace commit with the commit id hash of the latest tip of branch_9x)

On Sat, 29 Jun, 2024, 12:32 am Ishan Chattopadhyaya, <
ichattopadhyaya@gmail.com> wrote:

> Hi David,
> I'll publish benchmarks and steps to reproduce tests we are doing using
> solr-bench. Let's collaborate a bit more to make sure we can extract
> maximum performance out of the feature. Right now, I'm not seeing good
> numbers and it is a bit worrisome.
> Thanks and regards,
> Ishan
>
> On Sat, 29 Jun, 2024, 12:14 am Gus Heck, <gu...@gmail.com> wrote:
>
>> To comment further, I think it would be good if we had query benchmarks
>> that tried to map directly to the queries benchmarked for lucene, against
>> the same data. This would give us a notion of which half of the equation
>> any slow down comes from (or speed up!)
>>
>> On Fri, Jun 28, 2024 at 2:42 PM Gus Heck <gu...@gmail.com> wrote:
>>
>> > Yes. In fact I have an example in JesterJ of indexing the luceneutil
>> > data... (but it's still in early stages, I think it's still against
>> > _default schema, perhaps... (need to look again, haven't had time to
>> work
>> > on it recently)
>> >
>> > https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs
>> >
>> > On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org>
>> wrote:
>> >
>> >> I was thinking of using Solr's "benchmark" module/thing to benchmark
>> >> parallel segment search (coming to Solr 9.7 but needs more love).  I
>> >> don't notice any substantial data to query for in this module,
>> >> however.  Has anyone considered adding wikipedia, like how Lucene's
>> >> "luceneutil" does?  Or something else?  Is this a bad idea for this
>> >> benchmark module or should I be looking elsewhere like Searchscale's
>> >> solr-bench[1]?
>> >>
>> >> https://github.com/searchscale/solr-bench
>> >>
>> >> ~ David Smiley
>> >> Apache Lucene/Solr Search Developer
>> >> http://www.linkedin.com/in/davidwsmiley
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> >> For additional commands, e-mail: dev-help@solr.apache.org
>> >>
>> >>
>> >
>> > --
>> > http://www.needhamsoftware.com (work)
>> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
>> >
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>>
>

Re: Solr "benchmark", wikipedia

Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Hi David,
I'll publish benchmarks and steps to reproduce tests we are doing using
solr-bench. Let's collaborate a bit more to make sure we can extract
maximum performance out of the feature. Right now, I'm not seeing good
numbers and it is a bit worrisome.
Thanks and regards,
Ishan

On Sat, 29 Jun, 2024, 12:14 am Gus Heck, <gu...@gmail.com> wrote:

> To comment further, I think it would be good if we had query benchmarks
> that tried to map directly to the queries benchmarked for lucene, against
> the same data. This would give us a notion of which half of the equation
> any slow down comes from (or speed up!)
>
> On Fri, Jun 28, 2024 at 2:42 PM Gus Heck <gu...@gmail.com> wrote:
>
> > Yes. In fact I have an example in JesterJ of indexing the luceneutil
> > data... (but it's still in early stages, I think it's still against
> > _default schema, perhaps... (need to look again, haven't had time to work
> > on it recently)
> >
> > https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs
> >
> > On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org>
> wrote:
> >
> >> I was thinking of using Solr's "benchmark" module/thing to benchmark
> >> parallel segment search (coming to Solr 9.7 but needs more love).  I
> >> don't notice any substantial data to query for in this module,
> >> however.  Has anyone considered adding wikipedia, like how Lucene's
> >> "luceneutil" does?  Or something else?  Is this a bad idea for this
> >> benchmark module or should I be looking elsewhere like Searchscale's
> >> solr-bench[1]?
> >>
> >> https://github.com/searchscale/solr-bench
> >>
> >> ~ David Smiley
> >> Apache Lucene/Solr Search Developer
> >> http://www.linkedin.com/in/davidwsmiley
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >>
> >
> > --
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>

Re: Solr "benchmark", wikipedia

Posted by Gus Heck <gu...@gmail.com>.
To comment further, I think it would be good if we had query benchmarks
that tried to map directly to the queries benchmarked for lucene, against
the same data. This would give us a notion of which half of the equation
any slow down comes from (or speed up!)

On Fri, Jun 28, 2024 at 2:42 PM Gus Heck <gu...@gmail.com> wrote:

> Yes. In fact I have an example in JesterJ of indexing the luceneutil
> data... (but it's still in early stages, I think it's still against
> _default schema, perhaps... (need to look again, haven't had time to work
> on it recently)
>
> https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs
>
> On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org> wrote:
>
>> I was thinking of using Solr's "benchmark" module/thing to benchmark
>> parallel segment search (coming to Solr 9.7 but needs more love).  I
>> don't notice any substantial data to query for in this module,
>> however.  Has anyone considered adding wikipedia, like how Lucene's
>> "luceneutil" does?  Or something else?  Is this a bad idea for this
>> benchmark module or should I be looking elsewhere like Searchscale's
>> solr-bench[1]?
>>
>> https://github.com/searchscale/solr-bench
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>


-- 
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)

Re: Solr "benchmark", wikipedia

Posted by Gus Heck <gu...@gmail.com>.
Yes. In fact I have an example in JesterJ of indexing the luceneutil
data... (but it's still in early stages, I think it's still against
_default schema, perhaps... (need to look again, haven't had time to work
on it recently)

https://github.com/nsoft/jesterj/tree/master/code/examples/wikidocs

On Fri, Jun 28, 2024 at 11:33 AM David Smiley <ds...@apache.org> wrote:

> I was thinking of using Solr's "benchmark" module/thing to benchmark
> parallel segment search (coming to Solr 9.7 but needs more love).  I
> don't notice any substantial data to query for in this module,
> however.  Has anyone considered adding wikipedia, like how Lucene's
> "luceneutil" does?  Or something else?  Is this a bad idea for this
> benchmark module or should I be looking elsewhere like Searchscale's
> solr-bench[1]?
>
> https://github.com/searchscale/solr-bench
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>

-- 
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)