You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Jan Høydahl <ja...@cominvent.com> on 2026/04/16 07:36:00 UTC
Re: Solr Performance Benchmarking Discussion Page
Hi,
Thanks for presenting the solr-benchmark tool on the community meetup yesterday Kevin.
I'll encourage others to take it for a spin. There are rough edges but you should be able to get some results.
We also discussed the potential for moving the tool from my github space to asf.
That would make it easier for the community to contribute and collaborate. But it requires 2-3 PMC members interested in being maintainers.
If we make it an asf repo, David noted that we are not required to publish official releases until we feel compelled to do so.
Also, David noted the naming collision with the internal solr-benchmark module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race track) but in our Solar-system universe :)
As a next step I'll start a VOTE thread for accepting the two repos into ASF. The results of the VOTE will guide further action.
Jan
> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>
> Hi,
>
> I read through the benchmarking wiki page today. It was well written, thanks Kevin for reviving this effort, and in such a structured way!
>
> There's a renewed energy around benchmarking, and more tooling options than ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr Benchmark":
>
> https://github.com/janhoy/solr-benchmark
>
> You may quickly notice that this looks familiar. And yes, it is a fork/port of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters, using the same datasets and "workload"s as those tools.
> I first tried to start a fork 2 or 3 years ago but it stranded. I made a new effort using LLM agents and this first working version was prepared in a few afternoons.
> Even if the foundation is solid and proven over many years, this initial port is not complete, view it as a MVP and WIP. Only one workload / dataset is ported so far. Take it for a spin...
>
> I'll defer to Kevin to add it to the tools list of the wiki and continue the analysis effort with this as one of the contenders.
>
> Jan
>
>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) <kl...@bloomberg.net>:
>>
>> Hello everyone,
>>
>> Given the recent interest and discussion around Solr performance benchmarking, I figured it would be useful to 1) centralize the discussion and 2) bring it to a long-lived format (that's not email). So with that, I have started https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking (I figured there's still more discussion to be had before it becomes a SIP with technical requirements).
>>
>> I encourage anyone and everyone who is interested to provide their input (comment or edit). This is a community initiative, and shouldn't be limited by me or any biases I may have. Hopefully people find this useful in moving the discussion forward.
>>
>> -Kevin
>
Re: Solr Performance Benchmarking Discussion Page
Posted by "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" <de...@solr.apache.org>.
Wow, looks stunning
From: dev@solr.apache.org At: 04/20/26 13:06:10 UTC-4:00To: dev@solr.apache.org
Subject: Re: Solr Performance Benchmarking Discussion Page
I asked Claude to envision what a benchmark results page for Solr might look
like, and here is what he came up with:
http://www.cominvent.com/pub/solr-benchmarks3.html
This is 100% MOCKUP, a single self-contained HTML file, no real benchmarks run.
But note how the page is still interactive, lets you zoom in on results, filter
benchmarks etc :)
My thinking is, we can build a test-runner that is scheduled by
Jenkins/GithubWorkflow every night. Each run will provision some EC2 nodes, run
a number of benchmarks on branch_10x and branch_9x, tear down the nodes and
dump result json files into an S3 bucket. Then post-process to build a results
site like the above mock, published under e.g.
https://solr.apache.org/benchmarks. My company would be willing to sponsor the
AWS bill for the first year, then perhaps someone else could take over.
Jan
Re: Solr Performance Benchmarking Discussion Page
Posted by Jan Høydahl <ja...@cominvent.com>.
I asked Claude to envision what a benchmark results page for Solr might look like, and here is what he came up with:
http://www.cominvent.com/pub/solr-benchmarks3.html
This is 100% MOCKUP, a single self-contained HTML file, no real benchmarks run. But note how the page is still interactive, lets you zoom in on results, filter benchmarks etc :)
My thinking is, we can build a test-runner that is scheduled by Jenkins/GithubWorkflow every night. Each run will provision some EC2 nodes, run a number of benchmarks on branch_10x and branch_9x, tear down the nodes and dump result json files into an S3 bucket. Then post-process to build a results site like the above mock, published under e.g. https://solr.apache.org/benchmarks. My company would be willing to sponsor the AWS bill for the first year, then perhaps someone else could take over.
Jan
Re: Solr Performance Benchmarking Discussion Page
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
We have good steam in the solr-orbit repositories. But some more work to do before we have a stable product.
Feel free to jump in and test out the tool and work on open issues.
In particular I need a review on https://issues.apache.org/jira/browse/SOLR-18259 which is the IP clearance that we as a PMC does ourselves to make sure the codebase is good wrt copyright and licensing. I have filled in a form, but need others to review so we can file it with the incubator for official record keeping.
Jan
> 22. mai 2026 kl. 14:28 skrev Jan Høydahl <ja...@cominvent.com>:
>
> Both the solr-orbit and solr-orbit-workloads repositories are created and initialized with WIP code.
>
> I have enabled GH issues for both repos and added some initial issues
> - https://github.com/apache/solr-orbit/issues
> - https://github.com/apache/solr-orbit-workloads/issues
>
> The ubmrella incubation JIRA is https://issues.apache.org/jira/browse/SOLR-18253
>
> Feel free to jump in and contribute anywhere you like.
>
> Jan
>
>> 21. mai 2026 kl. 19:58 skrev Jan Høydahl <ja...@cominvent.com>:
>>
>> The VOTE thread has passed.
>>
>> I'll continue provisioning the solr-orbit and solr-orbit-workloads git repositories and push the baseline code.
>> We can then create a JIRA umbrella and sub tasks for all the other tasks needed.
>>
>> Jan
>>
>>> 1. mai 2026 kl. 15:37 skrev Eric Pugh <ep...@apache.org>:
>>>
>>> I want to share with the community that earlier this week I had the opportunity to visit Kevin and team and talk about Solr Orbit fitting thier specific use case, and I think we have consensus that this solution is how we want to move forward. I know it will solve my client's needs as well.
>>>
>>> I checked in with Jan, and I think the next step is to get a vote thread on bringing in this codebase as a `solr-orbit` subproject.
>>>
>>> On 2026/04/23 20:42:30 Eric Pugh wrote:
>>>> Home after some travel...
>>>>
>>>> I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
>>>>
>>>> I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed.
>>>>
>>>> On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote:
>>>>> "Solr Orbit" seems a fine name to me.
>>>>>
>>>>> Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
>>>>>
>>>>> I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
>>>>>
>>>>> The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
>>>>>
>>>>> In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
>>>>>
>>>>> Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
>>>>>
>>>>> From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
>>>>> Subject: Re: Solr Performance Benchmarking Discussion Page
>>>>>
>>>>> It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
>>>>> are known to pursue such violations. And I believe it is a violation of the
>>>>> Apache license too. I like "Solr Orbit" :)
>>>>>
>>>>> I'll wait some time to let all voices be heard wrt the various tools and which
>>>>> to pick going forward, then if everyone rallies (pun intended) around
>>>>> solr-benchmark, then we can proceed.
>>>>>
>>>>> Jan
>>>>>
>>>>>> 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
>>>>>>
>>>>>> Why avoid the word "Rally" if that's its foundation/lineage? If we
>>>>>> want to avoid risk of using that word, then we simply ask
>>>>>> ElasticSearch with our proposed name for our fork that includes that
>>>>>> word.
>>>>>>
>>>>>> On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks for presenting the solr-benchmark tool on the community meetup
>>>>> yesterday Kevin.
>>>>>>> I'll encourage others to take it for a spin. There are rough edges but you
>>>>> should be able to get some results.
>>>>>>>
>>>>>>> We also discussed the potential for moving the tool from my github space to
>>>>> asf.
>>>>>>> That would make it easier for the community to contribute and collaborate.
>>>>> But it requires 2-3 PMC members interested in being maintainers.
>>>>>>> If we make it an asf repo, David noted that we are not required to publish
>>>>> official releases until we feel compelled to do so.
>>>>>>> Also, David noted the naming collision with the internal solr-benchmark
>>>>> module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
>>>>> track) but in our Solar-system universe :)
>>>>>>>
>>>>>>> As a next step I'll start a VOTE thread for accepting the two repos into
>>>>> ASF. The results of the VOTE will guide further action.
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I read through the benchmarking wiki page today. It was well written,
>>>>> thanks Kevin for reviving this effort, and in such a structured way!
>>>>>>>>
>>>>>>>> There's a renewed energy around benchmarking, and more tooling options than
>>>>> ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
>>>>> Benchmark":
>>>>>>>>
>>>>>>>> https://github.com/janhoy/solr-benchmark
>>>>>>>>
>>>>>>>> You may quickly notice that this looks familiar. And yes, it is a fork/port
>>>>> of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
>>>>> using the same datasets and "workload"s as those tools.
>>>>>>>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
>>>>> new effort using LLM agents and this first working version was prepared in a
>>>>> few afternoons.
>>>>>>>> Even if the foundation is solid and proven over many years, this initial
>>>>> port is not complete, view it as a MVP and WIP. Only one workload / dataset is
>>>>> ported so far. Take it for a spin...
>>>>>>>>
>>>>>>>> I'll defer to Kevin to add it to the tools list of the wiki and continue
>>>>> the analysis effort with this as one of the contenders.
>>>>>>>>
>>>>>>>> Jan
>>>>>>>>
>>>>>>>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
>>>>> <kl...@bloomberg.net>:
>>>>>>>>>
>>>>>>>>> Hello everyone,
>>>>>>>>>
>>>>>>>>> Given the recent interest and discussion around Solr performance
>>>>> benchmarking, I figured it would be useful to 1) centralize the discussion and
>>>>> 2) bring it to a long-lived format (that's not email). So with that, I have
>>>>> started
>>>>> https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
>>>>> (I figured there's still more discussion to be had before it becomes a SIP with
>>>>> technical requirements).
>>>>>>>>>
>>>>>>>>> I encourage anyone and everyone who is interested to provide their input
>>>>> (comment or edit). This is a community initiative, and shouldn't be limited by
>>>>> me or any biases I may have. Hopefully people find this useful in moving the
>>>>> discussion forward.
>>>>>>>>>
>>>>>>>>> -Kevin
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by Jan Høydahl <ja...@cominvent.com>.
Both the solr-orbit and solr-orbit-workloads repositories are created and initialized with WIP code.
I have enabled GH issues for both repos and added some initial issues
- https://github.com/apache/solr-orbit/issues
- https://github.com/apache/solr-orbit-workloads/issues
The ubmrella incubation JIRA is https://issues.apache.org/jira/browse/SOLR-18253
Feel free to jump in and contribute anywhere you like.
Jan
> 21. mai 2026 kl. 19:58 skrev Jan Høydahl <ja...@cominvent.com>:
>
> The VOTE thread has passed.
>
> I'll continue provisioning the solr-orbit and solr-orbit-workloads git repositories and push the baseline code.
> We can then create a JIRA umbrella and sub tasks for all the other tasks needed.
>
> Jan
>
>> 1. mai 2026 kl. 15:37 skrev Eric Pugh <ep...@apache.org>:
>>
>> I want to share with the community that earlier this week I had the opportunity to visit Kevin and team and talk about Solr Orbit fitting thier specific use case, and I think we have consensus that this solution is how we want to move forward. I know it will solve my client's needs as well.
>>
>> I checked in with Jan, and I think the next step is to get a vote thread on bringing in this codebase as a `solr-orbit` subproject.
>>
>> On 2026/04/23 20:42:30 Eric Pugh wrote:
>>> Home after some travel...
>>>
>>> I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
>>>
>>> I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed.
>>>
>>> On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote:
>>>> "Solr Orbit" seems a fine name to me.
>>>>
>>>> Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
>>>>
>>>> I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
>>>>
>>>> The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
>>>>
>>>> In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
>>>>
>>>> Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
>>>>
>>>> From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
>>>> Subject: Re: Solr Performance Benchmarking Discussion Page
>>>>
>>>> It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
>>>> are known to pursue such violations. And I believe it is a violation of the
>>>> Apache license too. I like "Solr Orbit" :)
>>>>
>>>> I'll wait some time to let all voices be heard wrt the various tools and which
>>>> to pick going forward, then if everyone rallies (pun intended) around
>>>> solr-benchmark, then we can proceed.
>>>>
>>>> Jan
>>>>
>>>>> 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
>>>>>
>>>>> Why avoid the word "Rally" if that's its foundation/lineage? If we
>>>>> want to avoid risk of using that word, then we simply ask
>>>>> ElasticSearch with our proposed name for our fork that includes that
>>>>> word.
>>>>>
>>>>> On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for presenting the solr-benchmark tool on the community meetup
>>>> yesterday Kevin.
>>>>>> I'll encourage others to take it for a spin. There are rough edges but you
>>>> should be able to get some results.
>>>>>>
>>>>>> We also discussed the potential for moving the tool from my github space to
>>>> asf.
>>>>>> That would make it easier for the community to contribute and collaborate.
>>>> But it requires 2-3 PMC members interested in being maintainers.
>>>>>> If we make it an asf repo, David noted that we are not required to publish
>>>> official releases until we feel compelled to do so.
>>>>>> Also, David noted the naming collision with the internal solr-benchmark
>>>> module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
>>>> track) but in our Solar-system universe :)
>>>>>>
>>>>>> As a next step I'll start a VOTE thread for accepting the two repos into
>>>> ASF. The results of the VOTE will guide further action.
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I read through the benchmarking wiki page today. It was well written,
>>>> thanks Kevin for reviving this effort, and in such a structured way!
>>>>>>>
>>>>>>> There's a renewed energy around benchmarking, and more tooling options than
>>>> ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
>>>> Benchmark":
>>>>>>>
>>>>>>> https://github.com/janhoy/solr-benchmark
>>>>>>>
>>>>>>> You may quickly notice that this looks familiar. And yes, it is a fork/port
>>>> of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
>>>> using the same datasets and "workload"s as those tools.
>>>>>>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
>>>> new effort using LLM agents and this first working version was prepared in a
>>>> few afternoons.
>>>>>>> Even if the foundation is solid and proven over many years, this initial
>>>> port is not complete, view it as a MVP and WIP. Only one workload / dataset is
>>>> ported so far. Take it for a spin...
>>>>>>>
>>>>>>> I'll defer to Kevin to add it to the tools list of the wiki and continue
>>>> the analysis effort with this as one of the contenders.
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
>>>> <kl...@bloomberg.net>:
>>>>>>>>
>>>>>>>> Hello everyone,
>>>>>>>>
>>>>>>>> Given the recent interest and discussion around Solr performance
>>>> benchmarking, I figured it would be useful to 1) centralize the discussion and
>>>> 2) bring it to a long-lived format (that's not email). So with that, I have
>>>> started
>>>> https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
>>>> (I figured there's still more discussion to be had before it becomes a SIP with
>>>> technical requirements).
>>>>>>>>
>>>>>>>> I encourage anyone and everyone who is interested to provide their input
>>>> (comment or edit). This is a community initiative, and shouldn't be limited by
>>>> me or any biases I may have. Hopefully people find this useful in moving the
>>>> discussion forward.
>>>>>>>>
>>>>>>>> -Kevin
>>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by Jan Høydahl <ja...@cominvent.com>.
The VOTE thread has passed.
I'll continue provisioning the solr-orbit and solr-orbit-workloads git repositories and push the baseline code.
We can then create a JIRA umbrella and sub tasks for all the other tasks needed.
Jan
> 1. mai 2026 kl. 15:37 skrev Eric Pugh <ep...@apache.org>:
>
> I want to share with the community that earlier this week I had the opportunity to visit Kevin and team and talk about Solr Orbit fitting thier specific use case, and I think we have consensus that this solution is how we want to move forward. I know it will solve my client's needs as well.
>
> I checked in with Jan, and I think the next step is to get a vote thread on bringing in this codebase as a `solr-orbit` subproject.
>
> On 2026/04/23 20:42:30 Eric Pugh wrote:
>> Home after some travel...
>>
>> I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
>>
>> I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed.
>>
>> On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote:
>>> "Solr Orbit" seems a fine name to me.
>>>
>>> Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
>>>
>>> I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
>>>
>>> The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
>>>
>>> In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
>>>
>>> Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
>>>
>>> From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
>>> Subject: Re: Solr Performance Benchmarking Discussion Page
>>>
>>> It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
>>> are known to pursue such violations. And I believe it is a violation of the
>>> Apache license too. I like "Solr Orbit" :)
>>>
>>> I'll wait some time to let all voices be heard wrt the various tools and which
>>> to pick going forward, then if everyone rallies (pun intended) around
>>> solr-benchmark, then we can proceed.
>>>
>>> Jan
>>>
>>>> 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
>>>>
>>>> Why avoid the word "Rally" if that's its foundation/lineage? If we
>>>> want to avoid risk of using that word, then we simply ask
>>>> ElasticSearch with our proposed name for our fork that includes that
>>>> word.
>>>>
>>>> On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks for presenting the solr-benchmark tool on the community meetup
>>> yesterday Kevin.
>>>>> I'll encourage others to take it for a spin. There are rough edges but you
>>> should be able to get some results.
>>>>>
>>>>> We also discussed the potential for moving the tool from my github space to
>>> asf.
>>>>> That would make it easier for the community to contribute and collaborate.
>>> But it requires 2-3 PMC members interested in being maintainers.
>>>>> If we make it an asf repo, David noted that we are not required to publish
>>> official releases until we feel compelled to do so.
>>>>> Also, David noted the naming collision with the internal solr-benchmark
>>> module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
>>> track) but in our Solar-system universe :)
>>>>>
>>>>> As a next step I'll start a VOTE thread for accepting the two repos into
>>> ASF. The results of the VOTE will guide further action.
>>>>>
>>>>> Jan
>>>>>
>>>>>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I read through the benchmarking wiki page today. It was well written,
>>> thanks Kevin for reviving this effort, and in such a structured way!
>>>>>>
>>>>>> There's a renewed energy around benchmarking, and more tooling options than
>>> ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
>>> Benchmark":
>>>>>>
>>>>>> https://github.com/janhoy/solr-benchmark
>>>>>>
>>>>>> You may quickly notice that this looks familiar. And yes, it is a fork/port
>>> of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
>>> using the same datasets and "workload"s as those tools.
>>>>>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
>>> new effort using LLM agents and this first working version was prepared in a
>>> few afternoons.
>>>>>> Even if the foundation is solid and proven over many years, this initial
>>> port is not complete, view it as a MVP and WIP. Only one workload / dataset is
>>> ported so far. Take it for a spin...
>>>>>>
>>>>>> I'll defer to Kevin to add it to the tools list of the wiki and continue
>>> the analysis effort with this as one of the contenders.
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
>>> <kl...@bloomberg.net>:
>>>>>>>
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> Given the recent interest and discussion around Solr performance
>>> benchmarking, I figured it would be useful to 1) centralize the discussion and
>>> 2) bring it to a long-lived format (that's not email). So with that, I have
>>> started
>>> https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
>>> (I figured there's still more discussion to be had before it becomes a SIP with
>>> technical requirements).
>>>>>>>
>>>>>>> I encourage anyone and everyone who is interested to provide their input
>>> (comment or edit). This is a community initiative, and shouldn't be limited by
>>> me or any biases I may have. Hopefully people find this useful in moving the
>>> discussion forward.
>>>>>>>
>>>>>>> -Kevin
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>>> For additional commands, e-mail: dev-help@solr.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by Eric Pugh <ep...@apache.org>.
I want to share with the community that earlier this week I had the opportunity to visit Kevin and team and talk about Solr Orbit fitting thier specific use case, and I think we have consensus that this solution is how we want to move forward. I know it will solve my client's needs as well.
I checked in with Jan, and I think the next step is to get a vote thread on bringing in this codebase as a `solr-orbit` subproject.
On 2026/04/23 20:42:30 Eric Pugh wrote:
> Home after some travel...
>
> I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
>
> I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed.
>
> On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote:
> > "Solr Orbit" seems a fine name to me.
> >
> > Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
> >
> > I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
> >
> > The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
> >
> > In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
> >
> > Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
> >
> > From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
> > Subject: Re: Solr Performance Benchmarking Discussion Page
> >
> > It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
> > are known to pursue such violations. And I believe it is a violation of the
> > Apache license too. I like "Solr Orbit" :)
> >
> > I'll wait some time to let all voices be heard wrt the various tools and which
> > to pick going forward, then if everyone rallies (pun intended) around
> > solr-benchmark, then we can proceed.
> >
> > Jan
> >
> > > 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
> > >
> > > Why avoid the word "Rally" if that's its foundation/lineage? If we
> > > want to avoid risk of using that word, then we simply ask
> > > ElasticSearch with our proposed name for our fork that includes that
> > > word.
> > >
> > > On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Thanks for presenting the solr-benchmark tool on the community meetup
> > yesterday Kevin.
> > >> I'll encourage others to take it for a spin. There are rough edges but you
> > should be able to get some results.
> > >>
> > >> We also discussed the potential for moving the tool from my github space to
> > asf.
> > >> That would make it easier for the community to contribute and collaborate.
> > But it requires 2-3 PMC members interested in being maintainers.
> > >> If we make it an asf repo, David noted that we are not required to publish
> > official releases until we feel compelled to do so.
> > >> Also, David noted the naming collision with the internal solr-benchmark
> > module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
> > track) but in our Solar-system universe :)
> > >>
> > >> As a next step I'll start a VOTE thread for accepting the two repos into
> > ASF. The results of the VOTE will guide further action.
> > >>
> > >> Jan
> > >>
> > >>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I read through the benchmarking wiki page today. It was well written,
> > thanks Kevin for reviving this effort, and in such a structured way!
> > >>>
> > >>> There's a renewed energy around benchmarking, and more tooling options than
> > ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
> > Benchmark":
> > >>>
> > >>> https://github.com/janhoy/solr-benchmark
> > >>>
> > >>> You may quickly notice that this looks familiar. And yes, it is a fork/port
> > of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
> > using the same datasets and "workload"s as those tools.
> > >>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
> > new effort using LLM agents and this first working version was prepared in a
> > few afternoons.
> > >>> Even if the foundation is solid and proven over many years, this initial
> > port is not complete, view it as a MVP and WIP. Only one workload / dataset is
> > ported so far. Take it for a spin...
> > >>>
> > >>> I'll defer to Kevin to add it to the tools list of the wiki and continue
> > the analysis effort with this as one of the contenders.
> > >>>
> > >>> Jan
> > >>>
> > >>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
> > <kl...@bloomberg.net>:
> > >>>>
> > >>>> Hello everyone,
> > >>>>
> > >>>> Given the recent interest and discussion around Solr performance
> > benchmarking, I figured it would be useful to 1) centralize the discussion and
> > 2) bring it to a long-lived format (that's not email). So with that, I have
> > started
> > https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
> > (I figured there's still more discussion to be had before it becomes a SIP with
> > technical requirements).
> > >>>>
> > >>>> I encourage anyone and everyone who is interested to provide their input
> > (comment or edit). This is a community initiative, and shouldn't be limited by
> > me or any biases I may have. Hopefully people find this useful in moving the
> > discussion forward.
> > >>>>
> > >>>> -Kevin
> > >>>
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by Eric Pugh <ep...@apache.org>.
Home after some travel...
I think the solr-benchmark-game needs a bit more marinating to see the true value. Also, Kevin and (???) did some great analysis on it and pointed out some big weaknesses to it's approach.
I'm onboard with the solr-orbit plan, and can see how I would use it with my client to establish the benchmarks needed.
On 2026/04/17 20:38:17 "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" wrote:
> "Solr Orbit" seems a fine name to me.
>
> Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
>
> I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
>
> The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
>
> In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
>
> Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
>
> From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
> Subject: Re: Solr Performance Benchmarking Discussion Page
>
> It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
> are known to pursue such violations. And I believe it is a violation of the
> Apache license too. I like "Solr Orbit" :)
>
> I'll wait some time to let all voices be heard wrt the various tools and which
> to pick going forward, then if everyone rallies (pun intended) around
> solr-benchmark, then we can proceed.
>
> Jan
>
> > 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
> >
> > Why avoid the word "Rally" if that's its foundation/lineage? If we
> > want to avoid risk of using that word, then we simply ask
> > ElasticSearch with our proposed name for our fork that includes that
> > word.
> >
> > On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
> >>
> >> Hi,
> >>
> >> Thanks for presenting the solr-benchmark tool on the community meetup
> yesterday Kevin.
> >> I'll encourage others to take it for a spin. There are rough edges but you
> should be able to get some results.
> >>
> >> We also discussed the potential for moving the tool from my github space to
> asf.
> >> That would make it easier for the community to contribute and collaborate.
> But it requires 2-3 PMC members interested in being maintainers.
> >> If we make it an asf repo, David noted that we are not required to publish
> official releases until we feel compelled to do so.
> >> Also, David noted the naming collision with the internal solr-benchmark
> module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
> track) but in our Solar-system universe :)
> >>
> >> As a next step I'll start a VOTE thread for accepting the two repos into
> ASF. The results of the VOTE will guide further action.
> >>
> >> Jan
> >>
> >>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
> >>>
> >>> Hi,
> >>>
> >>> I read through the benchmarking wiki page today. It was well written,
> thanks Kevin for reviving this effort, and in such a structured way!
> >>>
> >>> There's a renewed energy around benchmarking, and more tooling options than
> ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
> Benchmark":
> >>>
> >>> https://github.com/janhoy/solr-benchmark
> >>>
> >>> You may quickly notice that this looks familiar. And yes, it is a fork/port
> of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
> using the same datasets and "workload"s as those tools.
> >>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
> new effort using LLM agents and this first working version was prepared in a
> few afternoons.
> >>> Even if the foundation is solid and proven over many years, this initial
> port is not complete, view it as a MVP and WIP. Only one workload / dataset is
> ported so far. Take it for a spin...
> >>>
> >>> I'll defer to Kevin to add it to the tools list of the wiki and continue
> the analysis effort with this as one of the contenders.
> >>>
> >>> Jan
> >>>
> >>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
> <kl...@bloomberg.net>:
> >>>>
> >>>> Hello everyone,
> >>>>
> >>>> Given the recent interest and discussion around Solr performance
> benchmarking, I figured it would be useful to 1) centralize the discussion and
> 2) bring it to a long-lived format (that's not email). So with that, I have
> started
> https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
> (I figured there's still more discussion to be had before it becomes a SIP with
> technical requirements).
> >>>>
> >>>> I encourage anyone and everyone who is interested to provide their input
> (comment or edit). This is a community initiative, and shouldn't be limited by
> me or any biases I may have. Hopefully people find this useful in moving the
> discussion forward.
> >>>>
> >>>> -Kevin
> >>>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by "Kevin Liang (BLOOMBERG/ 919 3RD A) via dev" <de...@solr.apache.org>.
"Solr Orbit" seems a fine name to me.
Re: David's point on picking one of search-benchmark-game vs. solr-benchmark-fork
I'll add my opinion as I've used / made code changes to both and have provided this feedback to Eric / Jan.
The search-benchmark-game is nice for its simplicity and results presentation. It can quickly give you some data points between two versions of software. That being said, at best it seems geared more towards assessing a library (lucene) than a full distributed system (solr). Its feature set is limited and the methodology for executing queries / drawing statistical conclusions leaves something to be desired (can expand in detail if anyone wants to hear).
In contrast, solr-benchmark (OSB fork) is building off a feature set constructed over the better part of a decade. It's a much larger codebase and a more mature benchmarking framework with much greater capability to do different kinds of performance test workloads.
Personally I would go with solr-benchmark. Building a web interface on top (or hooking up a jenkins build pipeline) can be done. And like we said in the meeting, just getting people to run the tool independently and publishing their results, even as plain text on the wiki, would be a good first step win.
From: dev@solr.apache.org At: 04/16/26 13:00:14 UTC-4:00To: dev@solr.apache.org
Subject: Re: Solr Performance Benchmarking Discussion Page
It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They
are known to pursue such violations. And I believe it is a violation of the
Apache license too. I like "Solr Orbit" :)
I'll wait some time to let all voices be heard wrt the various tools and which
to pick going forward, then if everyone rallies (pun intended) around
solr-benchmark, then we can proceed.
Jan
> 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
>
> Why avoid the word "Rally" if that's its foundation/lineage? If we
> want to avoid risk of using that word, then we simply ask
> ElasticSearch with our proposed name for our fork that includes that
> word.
>
> On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Hi,
>>
>> Thanks for presenting the solr-benchmark tool on the community meetup
yesterday Kevin.
>> I'll encourage others to take it for a spin. There are rough edges but you
should be able to get some results.
>>
>> We also discussed the potential for moving the tool from my github space to
asf.
>> That would make it easier for the community to contribute and collaborate.
But it requires 2-3 PMC members interested in being maintainers.
>> If we make it an asf repo, David noted that we are not required to publish
official releases until we feel compelled to do so.
>> Also, David noted the naming collision with the internal solr-benchmark
module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race
track) but in our Solar-system universe :)
>>
>> As a next step I'll start a VOTE thread for accepting the two repos into
ASF. The results of the VOTE will guide further action.
>>
>> Jan
>>
>>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>
>>> Hi,
>>>
>>> I read through the benchmarking wiki page today. It was well written,
thanks Kevin for reviving this effort, and in such a structured way!
>>>
>>> There's a renewed energy around benchmarking, and more tooling options than
ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr
Benchmark":
>>>
>>> https://github.com/janhoy/solr-benchmark
>>>
>>> You may quickly notice that this looks familiar. And yes, it is a fork/port
of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters,
using the same datasets and "workload"s as those tools.
>>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a
new effort using LLM agents and this first working version was prepared in a
few afternoons.
>>> Even if the foundation is solid and proven over many years, this initial
port is not complete, view it as a MVP and WIP. Only one workload / dataset is
ported so far. Take it for a spin...
>>>
>>> I'll defer to Kevin to add it to the tools list of the wiki and continue
the analysis effort with this as one of the contenders.
>>>
>>> Jan
>>>
>>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A)
<kl...@bloomberg.net>:
>>>>
>>>> Hello everyone,
>>>>
>>>> Given the recent interest and discussion around Solr performance
benchmarking, I figured it would be useful to 1) centralize the discussion and
2) bring it to a long-lived format (that's not email). So with that, I have
started
https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking
(I figured there's still more discussion to be had before it becomes a SIP with
technical requirements).
>>>>
>>>> I encourage anyone and everyone who is interested to provide their input
(comment or edit). This is a community initiative, and shouldn't be limited by
me or any biases I may have. Hopefully people find this useful in moving the
discussion forward.
>>>>
>>>> -Kevin
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by Jan Høydahl <ja...@cominvent.com>.
It's not work using Elastic Inc's trademark "Rally" if we can avoid it. They are known to pursue such violations. And I believe it is a violation of the Apache license too. I like "Solr Orbit" :)
I'll wait some time to let all voices be heard wrt the various tools and which to pick going forward, then if everyone rallies (pun intended) around solr-benchmark, then we can proceed.
Jan
> 16. apr. 2026 kl. 14:36 skrev David Smiley <ds...@apache.org>:
>
> Why avoid the word "Rally" if that's its foundation/lineage? If we
> want to avoid risk of using that word, then we simply ask
> ElasticSearch with our proposed name for our fork that includes that
> word.
>
> On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Hi,
>>
>> Thanks for presenting the solr-benchmark tool on the community meetup yesterday Kevin.
>> I'll encourage others to take it for a spin. There are rough edges but you should be able to get some results.
>>
>> We also discussed the potential for moving the tool from my github space to asf.
>> That would make it easier for the community to contribute and collaborate. But it requires 2-3 PMC members interested in being maintainers.
>> If we make it an asf repo, David noted that we are not required to publish official releases until we feel compelled to do so.
>> Also, David noted the naming collision with the internal solr-benchmark module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race track) but in our Solar-system universe :)
>>
>> As a next step I'll start a VOTE thread for accepting the two repos into ASF. The results of the VOTE will guide further action.
>>
>> Jan
>>
>>> 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
>>>
>>> Hi,
>>>
>>> I read through the benchmarking wiki page today. It was well written, thanks Kevin for reviving this effort, and in such a structured way!
>>>
>>> There's a renewed energy around benchmarking, and more tooling options than ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr Benchmark":
>>>
>>> https://github.com/janhoy/solr-benchmark
>>>
>>> You may quickly notice that this looks familiar. And yes, it is a fork/port of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters, using the same datasets and "workload"s as those tools.
>>> I first tried to start a fork 2 or 3 years ago but it stranded. I made a new effort using LLM agents and this first working version was prepared in a few afternoons.
>>> Even if the foundation is solid and proven over many years, this initial port is not complete, view it as a MVP and WIP. Only one workload / dataset is ported so far. Take it for a spin...
>>>
>>> I'll defer to Kevin to add it to the tools list of the wiki and continue the analysis effort with this as one of the contenders.
>>>
>>> Jan
>>>
>>>> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) <kl...@bloomberg.net>:
>>>>
>>>> Hello everyone,
>>>>
>>>> Given the recent interest and discussion around Solr performance benchmarking, I figured it would be useful to 1) centralize the discussion and 2) bring it to a long-lived format (that's not email). So with that, I have started https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking (I figured there's still more discussion to be had before it becomes a SIP with technical requirements).
>>>>
>>>> I encourage anyone and everyone who is interested to provide their input (comment or edit). This is a community initiative, and shouldn't be limited by me or any biases I may have. Hopefully people find this useful in moving the discussion forward.
>>>>
>>>> -Kevin
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by David Smiley <ds...@apache.org>.
Why avoid the word "Rally" if that's its foundation/lineage? If we
want to avoid risk of using that word, then we simply ask
ElasticSearch with our proposed name for our fork that includes that
word.
On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> Hi,
>
> Thanks for presenting the solr-benchmark tool on the community meetup yesterday Kevin.
> I'll encourage others to take it for a spin. There are rough edges but you should be able to get some results.
>
> We also discussed the potential for moving the tool from my github space to asf.
> That would make it easier for the community to contribute and collaborate. But it requires 2-3 PMC members interested in being maintainers.
> If we make it an asf repo, David noted that we are not required to publish official releases until we feel compelled to do so.
> Also, David noted the naming collision with the internal solr-benchmark module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race track) but in our Solar-system universe :)
>
> As a next step I'll start a VOTE thread for accepting the two repos into ASF. The results of the VOTE will guide further action.
>
> Jan
>
> > 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
> >
> > Hi,
> >
> > I read through the benchmarking wiki page today. It was well written, thanks Kevin for reviving this effort, and in such a structured way!
> >
> > There's a renewed energy around benchmarking, and more tooling options than ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr Benchmark":
> >
> > https://github.com/janhoy/solr-benchmark
> >
> > You may quickly notice that this looks familiar. And yes, it is a fork/port of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters, using the same datasets and "workload"s as those tools.
> > I first tried to start a fork 2 or 3 years ago but it stranded. I made a new effort using LLM agents and this first working version was prepared in a few afternoons.
> > Even if the foundation is solid and proven over many years, this initial port is not complete, view it as a MVP and WIP. Only one workload / dataset is ported so far. Take it for a spin...
> >
> > I'll defer to Kevin to add it to the tools list of the wiki and continue the analysis effort with this as one of the contenders.
> >
> > Jan
> >
> >> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) <kl...@bloomberg.net>:
> >>
> >> Hello everyone,
> >>
> >> Given the recent interest and discussion around Solr performance benchmarking, I figured it would be useful to 1) centralize the discussion and 2) bring it to a long-lived format (that's not email). So with that, I have started https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking (I figured there's still more discussion to be had before it becomes a SIP with technical requirements).
> >>
> >> I encourage anyone and everyone who is interested to provide their input (comment or edit). This is a community initiative, and shouldn't be limited by me or any biases I may have. Hopefully people find this useful in moving the discussion forward.
> >>
> >> -Kevin
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Solr Performance Benchmarking Discussion Page
Posted by David Smiley <ds...@apache.org>.
I suppose bringing this repo into the ASF doesn't preclude Eric doing
similar with solr-benchmark-game. But it'd probably be better to
disuss the merits between those two options and decide between them
instead of having two repos for benchmarking.
On Thu, Apr 16, 2026 at 3:37 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
> Hi,
>
> Thanks for presenting the solr-benchmark tool on the community meetup yesterday Kevin.
> I'll encourage others to take it for a spin. There are rough edges but you should be able to get some results.
>
> We also discussed the potential for moving the tool from my github space to asf.
> That would make it easier for the community to contribute and collaborate. But it requires 2-3 PMC members interested in being maintainers.
> If we make it an asf repo, David noted that we are not required to publish official releases until we feel compelled to do so.
> Also, David noted the naming collision with the internal solr-benchmark module. How about "solr-orbit", which is a homage to "Rally" (orbiting the race track) but in our Solar-system universe :)
>
> As a next step I'll start a VOTE thread for accepting the two repos into ASF. The results of the VOTE will guide further action.
>
> Jan
>
> > 8. mars 2026 kl. 20:42 skrev Jan Høydahl <ja...@cominvent.com>:
> >
> > Hi,
> >
> > I read through the benchmarking wiki page today. It was well written, thanks Kevin for reviving this effort, and in such a structured way!
> >
> > There's a renewed energy around benchmarking, and more tooling options than ever. So to add to the mix I'm presenting yet another one 🤣🤣, "Solr Benchmark":
> >
> > https://github.com/janhoy/solr-benchmark
> >
> > You may quickly notice that this looks familiar. And yes, it is a fork/port of Rally/OpensearchBenchmark, ported to provision and benchmark Solr clusters, using the same datasets and "workload"s as those tools.
> > I first tried to start a fork 2 or 3 years ago but it stranded. I made a new effort using LLM agents and this first working version was prepared in a few afternoons.
> > Even if the foundation is solid and proven over many years, this initial port is not complete, view it as a MVP and WIP. Only one workload / dataset is ported so far. Take it for a spin...
> >
> > I'll defer to Kevin to add it to the tools list of the wiki and continue the analysis effort with this as one of the contenders.
> >
> > Jan
> >
> >> 2. mars 2026 kl. 17:13 skrev Kevin Liang (BLOOMBERG/ 919 3RD A) <kl...@bloomberg.net>:
> >>
> >> Hello everyone,
> >>
> >> Given the recent interest and discussion around Solr performance benchmarking, I figured it would be useful to 1) centralize the discussion and 2) bring it to a long-lived format (that's not email). So with that, I have started https://cwiki.apache.org/confluence/display/SOLR/Solr+Performance+Benchmarking (I figured there's still more discussion to be had before it becomes a SIP with technical requirements).
> >>
> >> I encourage anyone and everyone who is interested to provide their input (comment or edit). This is a community initiative, and shouldn't be limited by me or any biases I may have. Hopefully people find this useful in moving the discussion forward.
> >>
> >> -Kevin
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org