You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@otava.apache.org by Sarthak Singh <si...@gmail.com> on 2026/06/27 22:21:29 UTC
[CONTRIB] Improving the Otava website and documentation (Issue #111)
Hey folks,
I'm Sarthak, and I'm reaching out to the list for the first time.
I saw the open ticket regarding updates to the website (#111)
<https://github.com/apache/otava/issues/111> and wanted to take a swing at
it.
I've opened an initial PR here:
https://github.com/apache/otava-website/pull/13
This PR updates the front page by adding some visualizations to help build
intuition for newcomers. Please review it and let me know what you
think. I'm open to feedback, suggestions, or alternative ideas for the
landing page.
My high-level plan for the rest of the ticket includes:
1. Docs Overhaul (Cosmetic & Content): Right now, the docs make a lot of
sense if you already know what Otava is, but they lack a structured,
logical flow for a newcomer. I'd love to restructure them to better explain
the problem Otava solves and how it works under the hood.
2. Community Section: Update this to explicitly list the project
Committers and PMC members (meeting the incubator guidelines mentioned in
the ticket).
3. Blog Tab: Set up the framework for a blog section. If anyone already
has content or a draft they'd like to share, please reply here and I can
incorporate it. Otherwise, I can lay the groundwork so it's ready when the
community needs it
As I'm new to Otava, I'd love to use this thread to:
- Iterate on feedback for the current PR and the broader website plan
- Ask questions and learn more from the existing community members
Looking forward to working with you all!
Best,
Sarthak
Re: [CONTRIB] Improving the Otava website and documentation (Issue #111)
Posted by David Daly <dr...@gmail.com>.
I think this is the other paper I mentioned -- but it's behind a paywall
and I don't have access.
https://www.sciencedirect.com/science/article/abs/pii/S0167642323001508
On Sat, Jul 4, 2026 at 5:19 PM David Daly <dr...@gmail.com> wrote:
> Hi all! I've been lurking. This all sounds great and thank you all for
> the good work. I love seeing it.
>
> One more paper i would suggest is:
> Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings
> of the ACM/SPEC International Conference on Performance Engineering
> https://dl.acm.org/doi/10.1145/3427921.3450234
>
> Which was a followup to the original change point paper. It made the point
> that better detection led to more tests which led to more detection and the
> need for better detection.
>
> There will also hopefully be a spec-rg paper coming out sometime in the
> near future, rigorously evaluating data sets and change point detection
> algorithms for regression detection.
>
>
> Probably not worth including, but there were four papers submitted for a
> related data challenge
> https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track
> Including for completeness. See the digital library below and look for
> the data challenge section.
>
> https://dl.acm.org/doi/proceedings/10.1145/3491204
>
> One of those papers had a nice extension to a journal paper, but it will
> take me a bit to find. That one got far enough that i wanted to take
> things for to Mongo.
>
> On Tue, Jun 30, 2026, 11:31 AM Henrik Ingo <he...@nyrkio.com> wrote:
>
>> On Mon, Jun 29, 2026 at 6:28 AM Alexander Sorokoumov <
>> aleksandr.sorokoumov@gmail.com> wrote:
>>
>> > Your plan also makes sense to me at a high-level. Regarding the blog, I
>> do
>> > wonder if we can borrow some of the articles from Nyrkio's blog that are
>> > specifically about Apache Otava (e..g, [3]).
>>
>>
>> Yes of course, perhaps also
>>
>> https://blog.nyrkio.com/2025/05/08/welcome-apache-otava-incubating-project/
>> which could be the first post on an otava blog as it is my look back on
>> the
>> history of the project until we became an ASF project.
>>
>> Note that for the Playground demo, perhaps the demo video is more useful
>> than the blog post and one alternative is to repost the video but write a
>> new blog post around it.
>>
>>
>>
>> > Also, there have been a bunch
>> > of papers/talks about Otava/Hunter over the years. It would be great to
>> > have a single page where all of them are linked.
>> >
>> >
>> Yes!
>>
>> I seem to have unpublished a page where I used to keep such a list, but
>> might as well collect one here. Or put these in a github issue:
>>
>> The original ICPE and DBTest papers:
>>
>>
>> -
>>
>> Daly et.al. 2020: The Use of Change Point Detection to Identify
>> Software
>> Performance Regressions …
>> <https://dl.acm.org/doi/10.1145/3358960.3375791>
>> -
>>
>> Henrik Ingo & David Daly 2020: Automated system performance testing at
>> MongoDB <https://dl.acm.org/doi/10.1145/3395032.3395323>
>> -
>>
>> Fleming & Kolaczkowski et.al. 2023: Hunter: Using Change Point
>> Detection
>> to Hunt for Performance Regressions
>> <https://dl.acm.org/doi/10.1145/3578244.3583719>
>>
>>
>> Note: ACM tracks downloads, and is the official publisher of these anyway,
>> so I try to link to those even if Google and others might list the
>> corresponding Arxiv page first.
>>
>> This was the first use of change point detection outside of MongoDB after
>> we had published the first ICPE paper:
>>
>> https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe
>>
>> This presentation by Alex is very complete and IMO has been rather
>> influential as well:
>>
>> https://www.confluent.io/events/kafka-summit-london-2024/automating-speed-a-proven-approach-to-preventing-performance-regressions-in/
>>
>>
>>
>> Then there are some smaller publications, in smaller conferences or
>> rejected papers. I'm sure I'm not aware of all of them, but I'm (co)author
>> in
>> 8 Years of Optimizing Apache Otava: How disconnected open source
>> developers
>> took an algorithm from n3 to constant time
>> <https://arxiv.org/abs/2505.06758>
>> Detection of Performance Changes in MooBench Results Using Nyrkiö on
>> GitHub
>> Actions <https://arxiv.org/abs/2510.11310>,
>>
>> henrik
>>
>>
>> --
>> *nyrkio.com <http://nyrkio.com/>* ~ *Continuous Benchmarking as a
>> Service*
>>
>> Henrik Ingo, CEO
>> henrik@nyrkio.com LinkedIn:
>> www.linkedin.com/in/heingo
>> +358 40 569 7354 Twitter:
>> twitter.com/h_ingo
>>
>
Re: [CONTRIB] Improving the Otava website and documentation (Issue #111)
Posted by David Daly <dr...@gmail.com>.
Hi all! I've been lurking. This all sounds great and thank you all for the
good work. I love seeing it.
One more paper i would suggest is:
Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings
of the ACM/SPEC International Conference on Performance Engineering
https://dl.acm.org/doi/10.1145/3427921.3450234
Which was a followup to the original change point paper. It made the point
that better detection led to more tests which led to more detection and the
need for better detection.
There will also hopefully be a spec-rg paper coming out sometime in the
near future, rigorously evaluating data sets and change point detection
algorithms for regression detection.
Probably not worth including, but there were four papers submitted for a
related data challenge
https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track
Including for completeness. See the digital library below and look for the
data challenge section.
https://dl.acm.org/doi/proceedings/10.1145/3491204
One of those papers had a nice extension to a journal paper, but it will
take me a bit to find. That one got far enough that i wanted to take
things for to Mongo.
On Tue, Jun 30, 2026, 11:31 AM Henrik Ingo <he...@nyrkio.com> wrote:
> On Mon, Jun 29, 2026 at 6:28 AM Alexander Sorokoumov <
> aleksandr.sorokoumov@gmail.com> wrote:
>
> > Your plan also makes sense to me at a high-level. Regarding the blog, I
> do
> > wonder if we can borrow some of the articles from Nyrkio's blog that are
> > specifically about Apache Otava (e..g, [3]).
>
>
> Yes of course, perhaps also
> https://blog.nyrkio.com/2025/05/08/welcome-apache-otava-incubating-project/
> which could be the first post on an otava blog as it is my look back on the
> history of the project until we became an ASF project.
>
> Note that for the Playground demo, perhaps the demo video is more useful
> than the blog post and one alternative is to repost the video but write a
> new blog post around it.
>
>
>
> > Also, there have been a bunch
> > of papers/talks about Otava/Hunter over the years. It would be great to
> > have a single page where all of them are linked.
> >
> >
> Yes!
>
> I seem to have unpublished a page where I used to keep such a list, but
> might as well collect one here. Or put these in a github issue:
>
> The original ICPE and DBTest papers:
>
>
> -
>
> Daly et.al. 2020: The Use of Change Point Detection to Identify
> Software
> Performance Regressions …
> <https://dl.acm.org/doi/10.1145/3358960.3375791>
> -
>
> Henrik Ingo & David Daly 2020: Automated system performance testing at
> MongoDB <https://dl.acm.org/doi/10.1145/3395032.3395323>
> -
>
> Fleming & Kolaczkowski et.al. 2023: Hunter: Using Change Point
> Detection
> to Hunt for Performance Regressions
> <https://dl.acm.org/doi/10.1145/3578244.3583719>
>
>
> Note: ACM tracks downloads, and is the official publisher of these anyway,
> so I try to link to those even if Google and others might list the
> corresponding Arxiv page first.
>
> This was the first use of change point detection outside of MongoDB after
> we had published the first ICPE paper:
>
> https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe
>
> This presentation by Alex is very complete and IMO has been rather
> influential as well:
>
> https://www.confluent.io/events/kafka-summit-london-2024/automating-speed-a-proven-approach-to-preventing-performance-regressions-in/
>
>
>
> Then there are some smaller publications, in smaller conferences or
> rejected papers. I'm sure I'm not aware of all of them, but I'm (co)author
> in
> 8 Years of Optimizing Apache Otava: How disconnected open source developers
> took an algorithm from n3 to constant time
> <https://arxiv.org/abs/2505.06758>
> Detection of Performance Changes in MooBench Results Using Nyrkiö on GitHub
> Actions <https://arxiv.org/abs/2510.11310>,
>
> henrik
>
>
> --
> *nyrkio.com <http://nyrkio.com/>* ~ *Continuous Benchmarking as a Service*
>
> Henrik Ingo, CEO
> henrik@nyrkio.com LinkedIn:
> www.linkedin.com/in/heingo
> +358 40 569 7354 Twitter:
> twitter.com/h_ingo
>
Re: [CONTRIB] Improving the Otava website and documentation (Issue #111)
Posted by Henrik Ingo <he...@nyrkio.com>.
On Mon, Jun 29, 2026 at 6:28 AM Alexander Sorokoumov <
aleksandr.sorokoumov@gmail.com> wrote:
> Your plan also makes sense to me at a high-level. Regarding the blog, I do
> wonder if we can borrow some of the articles from Nyrkio's blog that are
> specifically about Apache Otava (e..g, [3]).
Yes of course, perhaps also
https://blog.nyrkio.com/2025/05/08/welcome-apache-otava-incubating-project/
which could be the first post on an otava blog as it is my look back on the
history of the project until we became an ASF project.
Note that for the Playground demo, perhaps the demo video is more useful
than the blog post and one alternative is to repost the video but write a
new blog post around it.
> Also, there have been a bunch
> of papers/talks about Otava/Hunter over the years. It would be great to
> have a single page where all of them are linked.
>
>
Yes!
I seem to have unpublished a page where I used to keep such a list, but
might as well collect one here. Or put these in a github issue:
The original ICPE and DBTest papers:
-
Daly et.al. 2020: The Use of Change Point Detection to Identify Software
Performance Regressions …
<https://dl.acm.org/doi/10.1145/3358960.3375791>
-
Henrik Ingo & David Daly 2020: Automated system performance testing at
MongoDB <https://dl.acm.org/doi/10.1145/3395032.3395323>
-
Fleming & Kolaczkowski et.al. 2023: Hunter: Using Change Point Detection
to Hunt for Performance Regressions
<https://dl.acm.org/doi/10.1145/3578244.3583719>
Note: ACM tracks downloads, and is the official publisher of these anyway,
so I try to link to those even if Google and others might list the
corresponding Arxiv page first.
This was the first use of change point detection outside of MongoDB after
we had published the first ICPE paper:
https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe
This presentation by Alex is very complete and IMO has been rather
influential as well:
https://www.confluent.io/events/kafka-summit-london-2024/automating-speed-a-proven-approach-to-preventing-performance-regressions-in/
Then there are some smaller publications, in smaller conferences or
rejected papers. I'm sure I'm not aware of all of them, but I'm (co)author
in
8 Years of Optimizing Apache Otava: How disconnected open source developers
took an algorithm from n3 to constant time
<https://arxiv.org/abs/2505.06758>
Detection of Performance Changes in MooBench Results Using Nyrkiö on GitHub
Actions <https://arxiv.org/abs/2510.11310>,
henrik
--
*nyrkio.com <http://nyrkio.com/>* ~ *Continuous Benchmarking as a Service*
Henrik Ingo, CEO
henrik@nyrkio.com LinkedIn:
www.linkedin.com/in/heingo
+358 40 569 7354 Twitter: twitter.com/h_ingo
Re: [CONTRIB] Improving the Otava website and documentation (Issue #111)
Posted by Alexander Sorokoumov <al...@gmail.com>.
Hey Sarthak!
Welcome to the Otava community! I believe that improving the project
website is one of the most important development areas for the project, as
mentioned in our status reports (e.g., [1], [2]).
Your PR with the landing page improvement looks very promising! I left a
detailed review there.
Your plan also makes sense to me at a high-level. Regarding the blog, I do
wonder if we can borrow some of the articles from Nyrkio's blog that are
specifically about Apache Otava (e..g, [3]). Also, there have been a bunch
of papers/talks about Otava/Hunter over the years. It would be great to
have a single page where all of them are linked.
The docs overhaul may feel somewhat overwhelming, which is one of the
reasons I personally did not get to it. When in doubt, I encourage you to
bias towards reaching out for suggestions and ask questions, no matter how
small or nuanced they seem to be.
Best,
Alex
1. https://cwiki.apache.org/confluence/display/INCUBATOR/June2026#otava
2. https://cwiki.apache.org/confluence/display/INCUBATOR/March2026#otava
3.
https://blog.nyrkio.com/2026/06/06/change-detection-otava-0-8-0-released-compare-algorithms-with-otava-playground-demo-app/
On Sun, Jun 28, 2026 at 8:10 AM Sarthak Singh <si...@gmail.com>
wrote:
> Hey folks,
>
>
> I'm Sarthak, and I'm reaching out to the list for the first time.
>
>
> I saw the open ticket regarding updates to the website (#111)
> <https://github.com/apache/otava/issues/111> and wanted to take a swing at
> it.
>
> I've opened an initial PR here:
> https://github.com/apache/otava-website/pull/13
>
>
> This PR updates the front page by adding some visualizations to help build
> intuition for newcomers. Please review it and let me know what you
> think. I'm open to feedback, suggestions, or alternative ideas for the
> landing page.
>
>
> My high-level plan for the rest of the ticket includes:
>
> 1. Docs Overhaul (Cosmetic & Content): Right now, the docs make a lot of
> sense if you already know what Otava is, but they lack a structured,
> logical flow for a newcomer. I'd love to restructure them to better
> explain
> the problem Otava solves and how it works under the hood.
> 2. Community Section: Update this to explicitly list the project
> Committers and PMC members (meeting the incubator guidelines mentioned
> in
> the ticket).
> 3. Blog Tab: Set up the framework for a blog section. If anyone already
> has content or a draft they'd like to share, please reply here and I can
> incorporate it. Otherwise, I can lay the groundwork so it's ready when
> the
> community needs it
>
> As I'm new to Otava, I'd love to use this thread to:
>
> - Iterate on feedback for the current PR and the broader website plan
> - Ask questions and learn more from the existing community members
>
> Looking forward to working with you all!
>
>
> Best,
>
> Sarthak
>