Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

hi all,

I saw that conbench.ursa.dev has been down and I had a need to set up
some continuous project benchmarks, and was interested in doing
development on Conbench (well, having my agents do development on
Conbench), and was interested in the following:

1) is there interest in migrating the historical Arrow conbench data
to a new server, has that been preserved somewhere? I'll probably
rewrite the conbench backend in Go and give it a client CLI for
submitting new data or querying old data.

2) are there other users of conbench (conbench/conbench) that anyone
is aware of? I'd be done doing in-situ development in that repository
or setting up a conbench-v2 project.

No particular urgency but if anyone has opinions let me know!

thanks,
Wes

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Ok, I see no harm in testing this out as long we don't cut the working
deployment off.
Let's coordinate offline.

A thing to note here is that the cloud account in question is funded
by AWS's credit donation to Arrow's community. So please raise objections
if any, otherwise we'll proceed with lazy consensus.


On Wed, Jun 24, 2026 at 12:35 AM Wes McKinney <we...@gmail.com> wrote:

> If you can give me access to the servers and cloud resources in question
> (ie the machines that currently run the benchmarks), I can implement the
> necessary code adaptations and test things working end to end, and stand up
> a parallel deployment of the application against RDS to enable better
> evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
> back to the community with a report once the implementation is closer to
> being able to “throw the switch”.
>
> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:
>
> > Current architecture is not optimal or modern, but its maintenance cost
> > is well known and currently manageable. But I'm happy with changes that
> > move us to a more maintainable state and am willing to assist with the
> > transition.
> >
> > On a personal note, I'm hesitant to sign up for maintaining a deployment
> > of software that's not built yet.
> > So I'm just curious about who "owns" Arrow's conbench deployment
> > if I cannot commit to it.
> >
> > Rok
> >
> > On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > I think the objective is to create a more modern foundation while also
> > > improving performance. I think this looks like:
> > >
> > > * faster, easier to develop and deploy backup (use Go or Rust to create
> > > static binaries: Go is good for backend services like this)
> > > * use modern web technologies versus generating pages with Jinja
> > templates
> > > * Use things like DuckDB and Parquet to scale result storage while
> > > improving performance
> > > * Add many more UI features to make the results most useful to
> > maintainers
> > >
> > > Like I said, I’m happy to fulfill feature requests and contribute
> > > development with agents, if it isn’t interesting I’m also fine to go my
> > own
> > > way.
> > >
> > > I think Buildkite is fine for job scheduling and management for now, I
> > > don’t think this system currently wants to own a task queue / durable
> > > execution state for workers, though it could grow this capability in
> the
> > > future (workers would have to poll the server for jobs to take).
> > >
> > > On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
> wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > I think the main question here is: what are we trying to do?
> > > >
> > > > Currently, the main operational issue with conbench is the slowness
> of
> > > > the web UI, due to the large database size and that it's not
> normalized
> > > > (some queries take much longer than they should).
> > > >
> > > > I can't speak about the maintenance / reliability aspects, though.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > > > > hi all,
> > > > >
> > > > > I saw that conbench.ursa.dev has been down and I had a need to set
> > up
> > > > > some continuous project benchmarks, and was interested in doing
> > > > > development on Conbench (well, having my agents do development on
> > > > > Conbench), and was interested in the following:
> > > > >
> > > > > 1) is there interest in migrating the historical Arrow conbench
> data
> > > > > to a new server, has that been preserved somewhere? I'll probably
> > > > > rewrite the conbench backend in Go and give it a client CLI for
> > > > > submitting new data or querying old data.
> > > > >
> > > > > 2) are there other users of conbench (conbench/conbench) that
> anyone
> > > > > is aware of? I'd be done doing in-situ development in that
> repository
> > > > > or setting up a conbench-v2 project.
> > > > >
> > > > > No particular urgency but if anyone has opinions let me know!
> > > > >
> > > > > thanks,
> > > > > Wes
> > > >
> > > >
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Hi Wes et al,

I did a bit of thinking on this, speciffically the schema design, and wrote
up a draft [1].
Further, I met Ralf (cced) at EuroPython and it turns out he's faced
similar issues, so I proposed we pool resources.
We started a weekly sync and a temporary repo [2] to collect user stories
and requirements before starting the actual implementation. (If anyone is
interested please let me know and I'll send you an invite.)

I expect that once we have an approximately complete agreement on
requirements, we will proceed to implement components. If possible I'd like
to start with conbench's storage layer.

> I’m working on but I’m at a point where I want to start making sweeping
changes to the UI and I don’t want to be disruptive if folks form Arrow
want to use it

This sounds great and we definitely want innovation and better ergonomics!

[1]
https://docs.google.com/document/d/1ki5Y3b-Fqd9XZpKYBErbmV_Ups4jrwgr7B40W76kTKY/edit?usp=sharing
[2] https://github.com/rok/benchx

Rok

On Wed, Aug 26, 2026 at 10:07 PM Wes McKinney <we...@gmail.com> wrote:

> Hi all,
>
> I wanted to see if there is any interested in engaging on this? If not, I
> can just go my own way and call the new codebase something other than
> Conbench (I am using it to benchmark agentsview.io and some other projects
> I’m working on but I’m at a point where I want to start making sweeping
> changes to the UI and I don’t want to be disruptive if folks form Arrow
> want to use it). I’d prefer to move toward a model where much of the data
> lives in Parquet files queried by DuckDB for example rather than Postgres
> being the system of record.
>
> We are in a weird time at the moment since software is so easy to make if
> you have a clear idea of what you want, but it requires human engagement:
> the agents won’t make anything good without an dialed-in operator in the
> loop.
>
> Thanks
> Wes
>
> On Wed, Jul 8, 2026 at 14:22 Rok Mihevc <ro...@gmail.com> wrote:
>
> > We're seeing old [1] and new [2] conbench time out. I'm looking into
> > backend optimization to see if we can resolve our main pain point before
> > proceeding.
> >
> > [1] https://conbench.arrow-dev.org/
> > [2] https://conbench-v2.arrow-dev.org/
> >
> > Rok
> >
> > On Wed, Jul 1, 2026 at 11:20 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Important things for me in conbench:
> > >
> > > * regression detection, which is quite solid (the algorithm has been
> > > tweaked a lot AFAIK):
> > > https://github.com/apache/arrow/runs/84305379739
> > >
> > > * benchmark result pages and especially the "compare with baseline run
> > > from fork point commit", e.g.
> > > https://conbench.arrow-dev.org/runs/457fe991307a42d786798829edbc29f9/
> > >
> > > * the comparison pages such as
> > >
> > >
> >
> https://conbench.arrow-dev.org/compare/runs/1b61ec2a2670462f8c788da25fda3ca6...4ab7d6720b8c4e698e643f35819745c7/
> > > ; currently they are quite slow to compute and often time out
> > >
> > > Note some UI improvements would be warranted (especially: better
> > > filtering and/or display for benchmark results), we can discuss that
> too.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 27/06/2026 à 03:24, Wes McKinney a écrit :
> > > > I can update the chart to be more in line with the old one.
> > > >
> > > > I haven't done a great deal of work to enhance the UI (a lot of this
> > > > development was unattended and based on a mandate to rebuild the
> > > > backend in Go and rebuild the frontend on modern web technology, but
> > > > not make substantial changes). Using modern web technology (vite /
> > > > svelte) means it is much easier to add new things and make them feel
> > > > polished and nice, so the question is really what would be useful to
> > > > have?
> > > >
> > > > On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <ro...@gmail.com>
> > wrote:
> > > >>
> > > >> It's night here so I appreciate the new darker palette.
> > > >> As for result readability I prefer the old interactive graph because
> > it
> > > has
> > > >> classified points, trendline and bands.
> > > >>
> > > >> old:
> > > >>
> > >
> >
> https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
> > > >> new::
> > > >>
> > >
> >
> https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
> > > >>
> > > >> Any new feature that you'd point out for improved ergonomics?
> > > >>
> > > >> Rok
> > > >>
> > > >>
> > > >> On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >>
> > > >>> TL;DR take a look at https://conbench-v2.arrow-dev.org.
> > > >>>
> > > >>> This was written by Codex to summarize the state of things:
> > > >>>
> > > >>> Hi all,
> > > >>>
> > > >>> I’ve pushed the Conbench v2 work for review in the following
> places:
> > > >>>
> > > >>> Conbench v2 app/server/docs:
> > > >>> https://github.com/conbench/conbench/tree/experimental-v2
> > > >>>
> > > >>> Buildkite, Terraform, and CI adapter work:
> > > >>>
> > https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
> > > >>>
> > > >>> Python benchmark payload work:
> > > >>> https://github.com/wesm/benchmarks/tree/v2-conbench-submit
> > > >>>
> > > >>> R benchmark payload work:
> > > >>> https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
> > > >>>
> > > >>> There is also a live read-only evaluator running here:
> > > >>>
> > > >>> https://conbench-v2.arrow-dev.org
> > > >>>
> > > >>> Temporary docs are published here:
> > > >>>
> > > >>> https://wesm.github.io/conbench-tmp/
> > > >>>
> > > >>> The goal of this work is to provide a concrete migration path for
> > > >>> Conbench v2 while preserving the existing production database
> schema.
> > > >>> The evaluator is intended for review and experimentation only; it
> > > >>> should not modify the production database.
> > > >>>
> > > >>> Please take a look at the app, docs, and workflow changes. The main
> > > >>> things to review are whether the new UI is useful for Arrow
> > > >>> maintainers, whether the benchmark reporting path is
> understandable,
> > > >>> and whether the migration approach looks practical for the existing
> > > >>> Arrow benchmarking workflows.
> > > >>>
> > > >>> On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> If you can give me access to the servers and cloud resources in
> > > question
> > > >>> (ie the machines that currently run the benchmarks), I can
> implement
> > > the
> > > >>> necessary code adaptations and test things working end to end, and
> > > stand up
> > > >>> a parallel deployment of the application against RDS to enable
> better
> > > >>> evaluation of the UI ergonomics. Perhaps we can coordinate offline
> > and
> > > come
> > > >>> back to the community with a report once the implementation is
> closer
> > > to
> > > >>> being able to “throw the switch”.
> > > >>>>
> > > >>>> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com>
> > > wrote:
> > > >>>>>
> > > >>>>> Current architecture is not optimal or modern, but its
> maintenance
> > > cost
> > > >>>>> is well known and currently manageable. But I'm happy with
> changes
> > > that
> > > >>>>> move us to a more maintainable state and am willing to assist
> with
> > > the
> > > >>>>> transition.
> > > >>>>>
> > > >>>>> On a personal note, I'm hesitant to sign up for maintaining a
> > > deployment
> > > >>>>> of software that's not built yet.
> > > >>>>> So I'm just curious about who "owns" Arrow's conbench deployment
> > > >>>>> if I cannot commit to it.
> > > >>>>>
> > > >>>>> Rok
> > > >>>>>
> > > >>>>> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> I think the objective is to create a more modern foundation
> while
> > > also
> > > >>>>>> improving performance. I think this looks like:
> > > >>>>>>
> > > >>>>>> * faster, easier to develop and deploy backup (use Go or Rust to
> > > >>> create
> > > >>>>>> static binaries: Go is good for backend services like this)
> > > >>>>>> * use modern web technologies versus generating pages with Jinja
> > > >>> templates
> > > >>>>>> * Use things like DuckDB and Parquet to scale result storage
> while
> > > >>>>>> improving performance
> > > >>>>>> * Add many more UI features to make the results most useful to
> > > >>> maintainers
> > > >>>>>>
> > > >>>>>> Like I said, I’m happy to fulfill feature requests and
> contribute
> > > >>>>>> development with agents, if it isn’t interesting I’m also fine
> to
> > go
> > > >>> my own
> > > >>>>>> way.
> > > >>>>>>
> > > >>>>>> I think Buildkite is fine for job scheduling and management for
> > > now, I
> > > >>>>>> don’t think this system currently wants to own a task queue /
> > > durable
> > > >>>>>> execution state for workers, though it could grow this
> capability
> > in
> > > >>> the
> > > >>>>>> future (workers would have to poll the server for jobs to take).
> > > >>>>>>
> > > >>>>>> On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <
> antoine@python.org>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> Hi,
> > > >>>>>>>
> > > >>>>>>> I think the main question here is: what are we trying to do?
> > > >>>>>>>
> > > >>>>>>> Currently, the main operational issue with conbench is the
> > slowness
> > > >>> of
> > > >>>>>>> the web UI, due to the large database size and that it's not
> > > >>> normalized
> > > >>>>>>> (some queries take much longer than they should).
> > > >>>>>>>
> > > >>>>>>> I can't speak about the maintenance / reliability aspects,
> > though.
> > > >>>>>>>
> > > >>>>>>> Regards
> > > >>>>>>>
> > > >>>>>>> Antoine.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > > >>>>>>>> hi all,
> > > >>>>>>>>
> > > >>>>>>>> I saw that conbench.ursa.dev has been down and I had a need
> to
> > > >>> set up
> > > >>>>>>>> some continuous project benchmarks, and was interested in
> doing
> > > >>>>>>>> development on Conbench (well, having my agents do development
> > on
> > > >>>>>>>> Conbench), and was interested in the following:
> > > >>>>>>>>
> > > >>>>>>>> 1) is there interest in migrating the historical Arrow
> conbench
> > > >>> data
> > > >>>>>>>> to a new server, has that been preserved somewhere? I'll
> > probably
> > > >>>>>>>> rewrite the conbench backend in Go and give it a client CLI
> for
> > > >>>>>>>> submitting new data or querying old data.
> > > >>>>>>>>
> > > >>>>>>>> 2) are there other users of conbench (conbench/conbench) that
> > > >>> anyone
> > > >>>>>>>> is aware of? I'd be done doing in-situ development in that
> > > >>> repository
> > > >>>>>>>> or setting up a conbench-v2 project.
> > > >>>>>>>>
> > > >>>>>>>> No particular urgency but if anyone has opinions let me know!
> > > >>>>>>>>
> > > >>>>>>>> thanks,
> > > >>>>>>>> Wes
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>
> > >
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

Hi all,

I wanted to see if there is any interested in engaging on this? If not, I
can just go my own way and call the new codebase something other than
Conbench (I am using it to benchmark agentsview.io and some other projects
I’m working on but I’m at a point where I want to start making sweeping
changes to the UI and I don’t want to be disruptive if folks form Arrow
want to use it). I’d prefer to move toward a model where much of the data
lives in Parquet files queried by DuckDB for example rather than Postgres
being the system of record.

We are in a weird time at the moment since software is so easy to make if
you have a clear idea of what you want, but it requires human engagement:
the agents won’t make anything good without an dialed-in operator in the
loop.

Thanks
Wes

On Wed, Jul 8, 2026 at 14:22 Rok Mihevc <ro...@gmail.com> wrote:

> We're seeing old [1] and new [2] conbench time out. I'm looking into
> backend optimization to see if we can resolve our main pain point before
> proceeding.
>
> [1] https://conbench.arrow-dev.org/
> [2] https://conbench-v2.arrow-dev.org/
>
> Rok
>
> On Wed, Jul 1, 2026 at 11:20 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Important things for me in conbench:
> >
> > * regression detection, which is quite solid (the algorithm has been
> > tweaked a lot AFAIK):
> > https://github.com/apache/arrow/runs/84305379739
> >
> > * benchmark result pages and especially the "compare with baseline run
> > from fork point commit", e.g.
> > https://conbench.arrow-dev.org/runs/457fe991307a42d786798829edbc29f9/
> >
> > * the comparison pages such as
> >
> >
> https://conbench.arrow-dev.org/compare/runs/1b61ec2a2670462f8c788da25fda3ca6...4ab7d6720b8c4e698e643f35819745c7/
> > ; currently they are quite slow to compute and often time out
> >
> > Note some UI improvements would be warranted (especially: better
> > filtering and/or display for benchmark results), we can discuss that too.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 27/06/2026 à 03:24, Wes McKinney a écrit :
> > > I can update the chart to be more in line with the old one.
> > >
> > > I haven't done a great deal of work to enhance the UI (a lot of this
> > > development was unattended and based on a mandate to rebuild the
> > > backend in Go and rebuild the frontend on modern web technology, but
> > > not make substantial changes). Using modern web technology (vite /
> > > svelte) means it is much easier to add new things and make them feel
> > > polished and nice, so the question is really what would be useful to
> > > have?
> > >
> > > On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <ro...@gmail.com>
> wrote:
> > >>
> > >> It's night here so I appreciate the new darker palette.
> > >> As for result readability I prefer the old interactive graph because
> it
> > has
> > >> classified points, trendline and bands.
> > >>
> > >> old:
> > >>
> >
> https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
> > >> new::
> > >>
> >
> https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
> > >>
> > >> Any new feature that you'd point out for improved ergonomics?
> > >>
> > >> Rok
> > >>
> > >>
> > >> On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>
> > >>> TL;DR take a look at https://conbench-v2.arrow-dev.org.
> > >>>
> > >>> This was written by Codex to summarize the state of things:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> I’ve pushed the Conbench v2 work for review in the following places:
> > >>>
> > >>> Conbench v2 app/server/docs:
> > >>> https://github.com/conbench/conbench/tree/experimental-v2
> > >>>
> > >>> Buildkite, Terraform, and CI adapter work:
> > >>>
> https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
> > >>>
> > >>> Python benchmark payload work:
> > >>> https://github.com/wesm/benchmarks/tree/v2-conbench-submit
> > >>>
> > >>> R benchmark payload work:
> > >>> https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
> > >>>
> > >>> There is also a live read-only evaluator running here:
> > >>>
> > >>> https://conbench-v2.arrow-dev.org
> > >>>
> > >>> Temporary docs are published here:
> > >>>
> > >>> https://wesm.github.io/conbench-tmp/
> > >>>
> > >>> The goal of this work is to provide a concrete migration path for
> > >>> Conbench v2 while preserving the existing production database schema.
> > >>> The evaluator is intended for review and experimentation only; it
> > >>> should not modify the production database.
> > >>>
> > >>> Please take a look at the app, docs, and workflow changes. The main
> > >>> things to review are whether the new UI is useful for Arrow
> > >>> maintainers, whether the benchmark reporting path is understandable,
> > >>> and whether the migration approach looks practical for the existing
> > >>> Arrow benchmarking workflows.
> > >>>
> > >>> On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>>>
> > >>>> If you can give me access to the servers and cloud resources in
> > question
> > >>> (ie the machines that currently run the benchmarks), I can implement
> > the
> > >>> necessary code adaptations and test things working end to end, and
> > stand up
> > >>> a parallel deployment of the application against RDS to enable better
> > >>> evaluation of the UI ergonomics. Perhaps we can coordinate offline
> and
> > come
> > >>> back to the community with a report once the implementation is closer
> > to
> > >>> being able to “throw the switch”.
> > >>>>
> > >>>> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>> Current architecture is not optimal or modern, but its maintenance
> > cost
> > >>>>> is well known and currently manageable. But I'm happy with changes
> > that
> > >>>>> move us to a more maintainable state and am willing to assist with
> > the
> > >>>>> transition.
> > >>>>>
> > >>>>> On a personal note, I'm hesitant to sign up for maintaining a
> > deployment
> > >>>>> of software that's not built yet.
> > >>>>> So I'm just curious about who "owns" Arrow's conbench deployment
> > >>>>> if I cannot commit to it.
> > >>>>>
> > >>>>> Rok
> > >>>>>
> > >>>>> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> I think the objective is to create a more modern foundation while
> > also
> > >>>>>> improving performance. I think this looks like:
> > >>>>>>
> > >>>>>> * faster, easier to develop and deploy backup (use Go or Rust to
> > >>> create
> > >>>>>> static binaries: Go is good for backend services like this)
> > >>>>>> * use modern web technologies versus generating pages with Jinja
> > >>> templates
> > >>>>>> * Use things like DuckDB and Parquet to scale result storage while
> > >>>>>> improving performance
> > >>>>>> * Add many more UI features to make the results most useful to
> > >>> maintainers
> > >>>>>>
> > >>>>>> Like I said, I’m happy to fulfill feature requests and contribute
> > >>>>>> development with agents, if it isn’t interesting I’m also fine to
> go
> > >>> my own
> > >>>>>> way.
> > >>>>>>
> > >>>>>> I think Buildkite is fine for job scheduling and management for
> > now, I
> > >>>>>> don’t think this system currently wants to own a task queue /
> > durable
> > >>>>>> execution state for workers, though it could grow this capability
> in
> > >>> the
> > >>>>>> future (workers would have to poll the server for jobs to take).
> > >>>>>>
> > >>>>>> On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
> > >>> wrote:
> > >>>>>>
> > >>>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> I think the main question here is: what are we trying to do?
> > >>>>>>>
> > >>>>>>> Currently, the main operational issue with conbench is the
> slowness
> > >>> of
> > >>>>>>> the web UI, due to the large database size and that it's not
> > >>> normalized
> > >>>>>>> (some queries take much longer than they should).
> > >>>>>>>
> > >>>>>>> I can't speak about the maintenance / reliability aspects,
> though.
> > >>>>>>>
> > >>>>>>> Regards
> > >>>>>>>
> > >>>>>>> Antoine.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > >>>>>>>> hi all,
> > >>>>>>>>
> > >>>>>>>> I saw that conbench.ursa.dev has been down and I had a need to
> > >>> set up
> > >>>>>>>> some continuous project benchmarks, and was interested in doing
> > >>>>>>>> development on Conbench (well, having my agents do development
> on
> > >>>>>>>> Conbench), and was interested in the following:
> > >>>>>>>>
> > >>>>>>>> 1) is there interest in migrating the historical Arrow conbench
> > >>> data
> > >>>>>>>> to a new server, has that been preserved somewhere? I'll
> probably
> > >>>>>>>> rewrite the conbench backend in Go and give it a client CLI for
> > >>>>>>>> submitting new data or querying old data.
> > >>>>>>>>
> > >>>>>>>> 2) are there other users of conbench (conbench/conbench) that
> > >>> anyone
> > >>>>>>>> is aware of? I'd be done doing in-situ development in that
> > >>> repository
> > >>>>>>>> or setting up a conbench-v2 project.
> > >>>>>>>>
> > >>>>>>>> No particular urgency but if anyone has opinions let me know!
> > >>>>>>>>
> > >>>>>>>> thanks,
> > >>>>>>>> Wes
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>
> >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

We're seeing old [1] and new [2] conbench time out. I'm looking into
backend optimization to see if we can resolve our main pain point before
proceeding.

[1] https://conbench.arrow-dev.org/
[2] https://conbench-v2.arrow-dev.org/

Rok

On Wed, Jul 1, 2026 at 11:20 AM Antoine Pitrou <an...@python.org> wrote:

>
> Important things for me in conbench:
>
> * regression detection, which is quite solid (the algorithm has been
> tweaked a lot AFAIK):
> https://github.com/apache/arrow/runs/84305379739
>
> * benchmark result pages and especially the "compare with baseline run
> from fork point commit", e.g.
> https://conbench.arrow-dev.org/runs/457fe991307a42d786798829edbc29f9/
>
> * the comparison pages such as
>
> https://conbench.arrow-dev.org/compare/runs/1b61ec2a2670462f8c788da25fda3ca6...4ab7d6720b8c4e698e643f35819745c7/
> ; currently they are quite slow to compute and often time out
>
> Note some UI improvements would be warranted (especially: better
> filtering and/or display for benchmark results), we can discuss that too.
>
> Regards
>
> Antoine.
>
>
> Le 27/06/2026 à 03:24, Wes McKinney a écrit :
> > I can update the chart to be more in line with the old one.
> >
> > I haven't done a great deal of work to enhance the UI (a lot of this
> > development was unattended and based on a mandate to rebuild the
> > backend in Go and rebuild the frontend on modern web technology, but
> > not make substantial changes). Using modern web technology (vite /
> > svelte) means it is much easier to add new things and make them feel
> > polished and nice, so the question is really what would be useful to
> > have?
> >
> > On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <ro...@gmail.com> wrote:
> >>
> >> It's night here so I appreciate the new darker palette.
> >> As for result readability I prefer the old interactive graph because it
> has
> >> classified points, trendline and bands.
> >>
> >> old:
> >>
> https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
> >> new::
> >>
> https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
> >>
> >> Any new feature that you'd point out for improved ergonomics?
> >>
> >> Rok
> >>
> >>
> >> On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com>
> wrote:
> >>
> >>> TL;DR take a look at https://conbench-v2.arrow-dev.org.
> >>>
> >>> This was written by Codex to summarize the state of things:
> >>>
> >>> Hi all,
> >>>
> >>> I’ve pushed the Conbench v2 work for review in the following places:
> >>>
> >>> Conbench v2 app/server/docs:
> >>> https://github.com/conbench/conbench/tree/experimental-v2
> >>>
> >>> Buildkite, Terraform, and CI adapter work:
> >>> https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
> >>>
> >>> Python benchmark payload work:
> >>> https://github.com/wesm/benchmarks/tree/v2-conbench-submit
> >>>
> >>> R benchmark payload work:
> >>> https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
> >>>
> >>> There is also a live read-only evaluator running here:
> >>>
> >>> https://conbench-v2.arrow-dev.org
> >>>
> >>> Temporary docs are published here:
> >>>
> >>> https://wesm.github.io/conbench-tmp/
> >>>
> >>> The goal of this work is to provide a concrete migration path for
> >>> Conbench v2 while preserving the existing production database schema.
> >>> The evaluator is intended for review and experimentation only; it
> >>> should not modify the production database.
> >>>
> >>> Please take a look at the app, docs, and workflow changes. The main
> >>> things to review are whether the new UI is useful for Arrow
> >>> maintainers, whether the benchmark reporting path is understandable,
> >>> and whether the migration approach looks practical for the existing
> >>> Arrow benchmarking workflows.
> >>>
> >>> On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com>
> wrote:
> >>>>
> >>>> If you can give me access to the servers and cloud resources in
> question
> >>> (ie the machines that currently run the benchmarks), I can implement
> the
> >>> necessary code adaptations and test things working end to end, and
> stand up
> >>> a parallel deployment of the application against RDS to enable better
> >>> evaluation of the UI ergonomics. Perhaps we can coordinate offline and
> come
> >>> back to the community with a report once the implementation is closer
> to
> >>> being able to “throw the switch”.
> >>>>
> >>>> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com>
> wrote:
> >>>>>
> >>>>> Current architecture is not optimal or modern, but its maintenance
> cost
> >>>>> is well known and currently manageable. But I'm happy with changes
> that
> >>>>> move us to a more maintainable state and am willing to assist with
> the
> >>>>> transition.
> >>>>>
> >>>>> On a personal note, I'm hesitant to sign up for maintaining a
> deployment
> >>>>> of software that's not built yet.
> >>>>> So I'm just curious about who "owns" Arrow's conbench deployment
> >>>>> if I cannot commit to it.
> >>>>>
> >>>>> Rok
> >>>>>
> >>>>> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> I think the objective is to create a more modern foundation while
> also
> >>>>>> improving performance. I think this looks like:
> >>>>>>
> >>>>>> * faster, easier to develop and deploy backup (use Go or Rust to
> >>> create
> >>>>>> static binaries: Go is good for backend services like this)
> >>>>>> * use modern web technologies versus generating pages with Jinja
> >>> templates
> >>>>>> * Use things like DuckDB and Parquet to scale result storage while
> >>>>>> improving performance
> >>>>>> * Add many more UI features to make the results most useful to
> >>> maintainers
> >>>>>>
> >>>>>> Like I said, I’m happy to fulfill feature requests and contribute
> >>>>>> development with agents, if it isn’t interesting I’m also fine to go
> >>> my own
> >>>>>> way.
> >>>>>>
> >>>>>> I think Buildkite is fine for job scheduling and management for
> now, I
> >>>>>> don’t think this system currently wants to own a task queue /
> durable
> >>>>>> execution state for workers, though it could grow this capability in
> >>> the
> >>>>>> future (workers would have to poll the server for jobs to take).
> >>>>>>
> >>>>>> On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
> >>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I think the main question here is: what are we trying to do?
> >>>>>>>
> >>>>>>> Currently, the main operational issue with conbench is the slowness
> >>> of
> >>>>>>> the web UI, due to the large database size and that it's not
> >>> normalized
> >>>>>>> (some queries take much longer than they should).
> >>>>>>>
> >>>>>>> I can't speak about the maintenance / reliability aspects, though.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>>
> >>>>>>> Antoine.
> >>>>>>>
> >>>>>>>
> >>>>>>> Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> >>>>>>>> hi all,
> >>>>>>>>
> >>>>>>>> I saw that conbench.ursa.dev has been down and I had a need to
> >>> set up
> >>>>>>>> some continuous project benchmarks, and was interested in doing
> >>>>>>>> development on Conbench (well, having my agents do development on
> >>>>>>>> Conbench), and was interested in the following:
> >>>>>>>>
> >>>>>>>> 1) is there interest in migrating the historical Arrow conbench
> >>> data
> >>>>>>>> to a new server, has that been preserved somewhere? I'll probably
> >>>>>>>> rewrite the conbench backend in Go and give it a client CLI for
> >>>>>>>> submitting new data or querying old data.
> >>>>>>>>
> >>>>>>>> 2) are there other users of conbench (conbench/conbench) that
> >>> anyone
> >>>>>>>> is aware of? I'd be done doing in-situ development in that
> >>> repository
> >>>>>>>> or setting up a conbench-v2 project.
> >>>>>>>>
> >>>>>>>> No particular urgency but if anyone has opinions let me know!
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Wes
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>
>
>

Re: Status of Arrow conbench data + conbench OSS project

[Antoine Pitrou <an...@python.org>]

Important things for me in conbench:

* regression detection, which is quite solid (the algorithm has been 
tweaked a lot AFAIK):
https://github.com/apache/arrow/runs/84305379739

* benchmark result pages and especially the "compare with baseline run 
from fork point commit", e.g. 
https://conbench.arrow-dev.org/runs/457fe991307a42d786798829edbc29f9/

* the comparison pages such as 
https://conbench.arrow-dev.org/compare/runs/1b61ec2a2670462f8c788da25fda3ca6...4ab7d6720b8c4e698e643f35819745c7/ 
; currently they are quite slow to compute and often time out

Note some UI improvements would be warranted (especially: better 
filtering and/or display for benchmark results), we can discuss that too.

Regards

Antoine.


Le 27/06/2026 à 03:24, Wes McKinney a écrit :
> I can update the chart to be more in line with the old one.
> 
> I haven't done a great deal of work to enhance the UI (a lot of this
> development was unattended and based on a mandate to rebuild the
> backend in Go and rebuild the frontend on modern web technology, but
> not make substantial changes). Using modern web technology (vite /
> svelte) means it is much easier to add new things and make them feel
> polished and nice, so the question is really what would be useful to
> have?
> 
> On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <ro...@gmail.com> wrote:
>>
>> It's night here so I appreciate the new darker palette.
>> As for result readability I prefer the old interactive graph because it has
>> classified points, trendline and bands.
>>
>> old:
>> https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
>> new::
>> https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
>>
>> Any new feature that you'd point out for improved ergonomics?
>>
>> Rok
>>
>>
>> On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com> wrote:
>>
>>> TL;DR take a look at https://conbench-v2.arrow-dev.org.
>>>
>>> This was written by Codex to summarize the state of things:
>>>
>>> Hi all,
>>>
>>> I’ve pushed the Conbench v2 work for review in the following places:
>>>
>>> Conbench v2 app/server/docs:
>>> https://github.com/conbench/conbench/tree/experimental-v2
>>>
>>> Buildkite, Terraform, and CI adapter work:
>>> https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
>>>
>>> Python benchmark payload work:
>>> https://github.com/wesm/benchmarks/tree/v2-conbench-submit
>>>
>>> R benchmark payload work:
>>> https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
>>>
>>> There is also a live read-only evaluator running here:
>>>
>>> https://conbench-v2.arrow-dev.org
>>>
>>> Temporary docs are published here:
>>>
>>> https://wesm.github.io/conbench-tmp/
>>>
>>> The goal of this work is to provide a concrete migration path for
>>> Conbench v2 while preserving the existing production database schema.
>>> The evaluator is intended for review and experimentation only; it
>>> should not modify the production database.
>>>
>>> Please take a look at the app, docs, and workflow changes. The main
>>> things to review are whether the new UI is useful for Arrow
>>> maintainers, whether the benchmark reporting path is understandable,
>>> and whether the migration approach looks practical for the existing
>>> Arrow benchmarking workflows.
>>>
>>> On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> If you can give me access to the servers and cloud resources in question
>>> (ie the machines that currently run the benchmarks), I can implement the
>>> necessary code adaptations and test things working end to end, and stand up
>>> a parallel deployment of the application against RDS to enable better
>>> evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
>>> back to the community with a report once the implementation is closer to
>>> being able to “throw the switch”.
>>>>
>>>> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:
>>>>>
>>>>> Current architecture is not optimal or modern, but its maintenance cost
>>>>> is well known and currently manageable. But I'm happy with changes that
>>>>> move us to a more maintainable state and am willing to assist with the
>>>>> transition.
>>>>>
>>>>> On a personal note, I'm hesitant to sign up for maintaining a deployment
>>>>> of software that's not built yet.
>>>>> So I'm just curious about who "owns" Arrow's conbench deployment
>>>>> if I cannot commit to it.
>>>>>
>>>>> Rok
>>>>>
>>>>> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>>>>
>>>>>> I think the objective is to create a more modern foundation while also
>>>>>> improving performance. I think this looks like:
>>>>>>
>>>>>> * faster, easier to develop and deploy backup (use Go or Rust to
>>> create
>>>>>> static binaries: Go is good for backend services like this)
>>>>>> * use modern web technologies versus generating pages with Jinja
>>> templates
>>>>>> * Use things like DuckDB and Parquet to scale result storage while
>>>>>> improving performance
>>>>>> * Add many more UI features to make the results most useful to
>>> maintainers
>>>>>>
>>>>>> Like I said, I’m happy to fulfill feature requests and contribute
>>>>>> development with agents, if it isn’t interesting I’m also fine to go
>>> my own
>>>>>> way.
>>>>>>
>>>>>> I think Buildkite is fine for job scheduling and management for now, I
>>>>>> don’t think this system currently wants to own a task queue / durable
>>>>>> execution state for workers, though it could grow this capability in
>>> the
>>>>>> future (workers would have to poll the server for jobs to take).
>>>>>>
>>>>>> On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I think the main question here is: what are we trying to do?
>>>>>>>
>>>>>>> Currently, the main operational issue with conbench is the slowness
>>> of
>>>>>>> the web UI, due to the large database size and that it's not
>>> normalized
>>>>>>> (some queries take much longer than they should).
>>>>>>>
>>>>>>> I can't speak about the maintenance / reliability aspects, though.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Antoine.
>>>>>>>
>>>>>>>
>>>>>>> Le 04/06/2026 à 16:19, Wes McKinney a écrit :
>>>>>>>> hi all,
>>>>>>>>
>>>>>>>> I saw that conbench.ursa.dev has been down and I had a need to
>>> set up
>>>>>>>> some continuous project benchmarks, and was interested in doing
>>>>>>>> development on Conbench (well, having my agents do development on
>>>>>>>> Conbench), and was interested in the following:
>>>>>>>>
>>>>>>>> 1) is there interest in migrating the historical Arrow conbench
>>> data
>>>>>>>> to a new server, has that been preserved somewhere? I'll probably
>>>>>>>> rewrite the conbench backend in Go and give it a client CLI for
>>>>>>>> submitting new data or querying old data.
>>>>>>>>
>>>>>>>> 2) are there other users of conbench (conbench/conbench) that
>>> anyone
>>>>>>>> is aware of? I'd be done doing in-situ development in that
>>> repository
>>>>>>>> or setting up a conbench-v2 project.
>>>>>>>>
>>>>>>>> No particular urgency but if anyone has opinions let me know!
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Wes
>>>>>>>
>>>>>>>
>>>>>>
>>>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

I can update the chart to be more in line with the old one.

I haven't done a great deal of work to enhance the UI (a lot of this
development was unattended and based on a mandate to rebuild the
backend in Go and rebuild the frontend on modern web technology, but
not make substantial changes). Using modern web technology (vite /
svelte) means it is much easier to add new things and make them feel
polished and nice, so the question is really what would be useful to
have?

On Fri, Jun 26, 2026 at 7:56 PM Rok Mihevc <ro...@gmail.com> wrote:
>
> It's night here so I appreciate the new darker palette.
> As for result readability I prefer the old interactive graph because it has
> classified points, trendline and bands.
>
> old:
> https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
> new::
> https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e
>
> Any new feature that you'd point out for improved ergonomics?
>
> Rok
>
>
> On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com> wrote:
>
> > TL;DR take a look at https://conbench-v2.arrow-dev.org.
> >
> > This was written by Codex to summarize the state of things:
> >
> > Hi all,
> >
> > I’ve pushed the Conbench v2 work for review in the following places:
> >
> > Conbench v2 app/server/docs:
> > https://github.com/conbench/conbench/tree/experimental-v2
> >
> > Buildkite, Terraform, and CI adapter work:
> > https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
> >
> > Python benchmark payload work:
> > https://github.com/wesm/benchmarks/tree/v2-conbench-submit
> >
> > R benchmark payload work:
> > https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
> >
> > There is also a live read-only evaluator running here:
> >
> > https://conbench-v2.arrow-dev.org
> >
> > Temporary docs are published here:
> >
> > https://wesm.github.io/conbench-tmp/
> >
> > The goal of this work is to provide a concrete migration path for
> > Conbench v2 while preserving the existing production database schema.
> > The evaluator is intended for review and experimentation only; it
> > should not modify the production database.
> >
> > Please take a look at the app, docs, and workflow changes. The main
> > things to review are whether the new UI is useful for Arrow
> > maintainers, whether the benchmark reporting path is understandable,
> > and whether the migration approach looks practical for the existing
> > Arrow benchmarking workflows.
> >
> > On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > If you can give me access to the servers and cloud resources in question
> > (ie the machines that currently run the benchmarks), I can implement the
> > necessary code adaptations and test things working end to end, and stand up
> > a parallel deployment of the application against RDS to enable better
> > evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
> > back to the community with a report once the implementation is closer to
> > being able to “throw the switch”.
> > >
> > > On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:
> > >>
> > >> Current architecture is not optimal or modern, but its maintenance cost
> > >> is well known and currently manageable. But I'm happy with changes that
> > >> move us to a more maintainable state and am willing to assist with the
> > >> transition.
> > >>
> > >> On a personal note, I'm hesitant to sign up for maintaining a deployment
> > >> of software that's not built yet.
> > >> So I'm just curious about who "owns" Arrow's conbench deployment
> > >> if I cannot commit to it.
> > >>
> > >> Rok
> > >>
> > >> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >>
> > >> > I think the objective is to create a more modern foundation while also
> > >> > improving performance. I think this looks like:
> > >> >
> > >> > * faster, easier to develop and deploy backup (use Go or Rust to
> > create
> > >> > static binaries: Go is good for backend services like this)
> > >> > * use modern web technologies versus generating pages with Jinja
> > templates
> > >> > * Use things like DuckDB and Parquet to scale result storage while
> > >> > improving performance
> > >> > * Add many more UI features to make the results most useful to
> > maintainers
> > >> >
> > >> > Like I said, I’m happy to fulfill feature requests and contribute
> > >> > development with agents, if it isn’t interesting I’m also fine to go
> > my own
> > >> > way.
> > >> >
> > >> > I think Buildkite is fine for job scheduling and management for now, I
> > >> > don’t think this system currently wants to own a task queue / durable
> > >> > execution state for workers, though it could grow this capability in
> > the
> > >> > future (workers would have to poll the server for jobs to take).
> > >> >
> > >> > On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
> > wrote:
> > >> >
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I think the main question here is: what are we trying to do?
> > >> > >
> > >> > > Currently, the main operational issue with conbench is the slowness
> > of
> > >> > > the web UI, due to the large database size and that it's not
> > normalized
> > >> > > (some queries take much longer than they should).
> > >> > >
> > >> > > I can't speak about the maintenance / reliability aspects, though.
> > >> > >
> > >> > > Regards
> > >> > >
> > >> > > Antoine.
> > >> > >
> > >> > >
> > >> > > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > >> > > > hi all,
> > >> > > >
> > >> > > > I saw that conbench.ursa.dev has been down and I had a need to
> > set up
> > >> > > > some continuous project benchmarks, and was interested in doing
> > >> > > > development on Conbench (well, having my agents do development on
> > >> > > > Conbench), and was interested in the following:
> > >> > > >
> > >> > > > 1) is there interest in migrating the historical Arrow conbench
> > data
> > >> > > > to a new server, has that been preserved somewhere? I'll probably
> > >> > > > rewrite the conbench backend in Go and give it a client CLI for
> > >> > > > submitting new data or querying old data.
> > >> > > >
> > >> > > > 2) are there other users of conbench (conbench/conbench) that
> > anyone
> > >> > > > is aware of? I'd be done doing in-situ development in that
> > repository
> > >> > > > or setting up a conbench-v2 project.
> > >> > > >
> > >> > > > No particular urgency but if anyone has opinions let me know!
> > >> > > >
> > >> > > > thanks,
> > >> > > > Wes
> > >> > >
> > >> > >
> > >> >
> >

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

It's night here so I appreciate the new darker palette.
As for result readability I prefer the old interactive graph because it has
classified points, trendline and bands.

old:
https://conbench.arrow-dev.org/benchmark-results/06a3edbc2d1f7dc48000633dc612769e/
new::
https://conbench-v2.arrow-dev.org/benchmarks/history/06a3edbc2d1f7dc48000633dc612769e

Any new feature that you'd point out for improved ergonomics?

Rok


On Sat, Jun 27, 2026 at 2:22 AM Wes McKinney <we...@gmail.com> wrote:

> TL;DR take a look at https://conbench-v2.arrow-dev.org.
>
> This was written by Codex to summarize the state of things:
>
> Hi all,
>
> I’ve pushed the Conbench v2 work for review in the following places:
>
> Conbench v2 app/server/docs:
> https://github.com/conbench/conbench/tree/experimental-v2
>
> Buildkite, Terraform, and CI adapter work:
> https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report
>
> Python benchmark payload work:
> https://github.com/wesm/benchmarks/tree/v2-conbench-submit
>
> R benchmark payload work:
> https://github.com/wesm/arrowbench/tree/v2-conbench-payloads
>
> There is also a live read-only evaluator running here:
>
> https://conbench-v2.arrow-dev.org
>
> Temporary docs are published here:
>
> https://wesm.github.io/conbench-tmp/
>
> The goal of this work is to provide a concrete migration path for
> Conbench v2 while preserving the existing production database schema.
> The evaluator is intended for review and experimentation only; it
> should not modify the production database.
>
> Please take a look at the app, docs, and workflow changes. The main
> things to review are whether the new UI is useful for Arrow
> maintainers, whether the benchmark reporting path is understandable,
> and whether the migration approach looks practical for the existing
> Arrow benchmarking workflows.
>
> On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > If you can give me access to the servers and cloud resources in question
> (ie the machines that currently run the benchmarks), I can implement the
> necessary code adaptations and test things working end to end, and stand up
> a parallel deployment of the application against RDS to enable better
> evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
> back to the community with a report once the implementation is closer to
> being able to “throw the switch”.
> >
> > On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:
> >>
> >> Current architecture is not optimal or modern, but its maintenance cost
> >> is well known and currently manageable. But I'm happy with changes that
> >> move us to a more maintainable state and am willing to assist with the
> >> transition.
> >>
> >> On a personal note, I'm hesitant to sign up for maintaining a deployment
> >> of software that's not built yet.
> >> So I'm just curious about who "owns" Arrow's conbench deployment
> >> if I cannot commit to it.
> >>
> >> Rok
> >>
> >> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com>
> wrote:
> >>
> >> > I think the objective is to create a more modern foundation while also
> >> > improving performance. I think this looks like:
> >> >
> >> > * faster, easier to develop and deploy backup (use Go or Rust to
> create
> >> > static binaries: Go is good for backend services like this)
> >> > * use modern web technologies versus generating pages with Jinja
> templates
> >> > * Use things like DuckDB and Parquet to scale result storage while
> >> > improving performance
> >> > * Add many more UI features to make the results most useful to
> maintainers
> >> >
> >> > Like I said, I’m happy to fulfill feature requests and contribute
> >> > development with agents, if it isn’t interesting I’m also fine to go
> my own
> >> > way.
> >> >
> >> > I think Buildkite is fine for job scheduling and management for now, I
> >> > don’t think this system currently wants to own a task queue / durable
> >> > execution state for workers, though it could grow this capability in
> the
> >> > future (workers would have to poll the server for jobs to take).
> >> >
> >> > On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org>
> wrote:
> >> >
> >> > >
> >> > > Hi,
> >> > >
> >> > > I think the main question here is: what are we trying to do?
> >> > >
> >> > > Currently, the main operational issue with conbench is the slowness
> of
> >> > > the web UI, due to the large database size and that it's not
> normalized
> >> > > (some queries take much longer than they should).
> >> > >
> >> > > I can't speak about the maintenance / reliability aspects, though.
> >> > >
> >> > > Regards
> >> > >
> >> > > Antoine.
> >> > >
> >> > >
> >> > > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> >> > > > hi all,
> >> > > >
> >> > > > I saw that conbench.ursa.dev has been down and I had a need to
> set up
> >> > > > some continuous project benchmarks, and was interested in doing
> >> > > > development on Conbench (well, having my agents do development on
> >> > > > Conbench), and was interested in the following:
> >> > > >
> >> > > > 1) is there interest in migrating the historical Arrow conbench
> data
> >> > > > to a new server, has that been preserved somewhere? I'll probably
> >> > > > rewrite the conbench backend in Go and give it a client CLI for
> >> > > > submitting new data or querying old data.
> >> > > >
> >> > > > 2) are there other users of conbench (conbench/conbench) that
> anyone
> >> > > > is aware of? I'd be done doing in-situ development in that
> repository
> >> > > > or setting up a conbench-v2 project.
> >> > > >
> >> > > > No particular urgency but if anyone has opinions let me know!
> >> > > >
> >> > > > thanks,
> >> > > > Wes
> >> > >
> >> > >
> >> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

TL;DR take a look at https://conbench-v2.arrow-dev.org.

This was written by Codex to summarize the state of things:

Hi all,

I’ve pushed the Conbench v2 work for review in the following places:

Conbench v2 app/server/docs:
https://github.com/conbench/conbench/tree/experimental-v2

Buildkite, Terraform, and CI adapter work:
https://github.com/wesm/arrow-benchmarks-ci/tree/v2-conbench-ci-report

Python benchmark payload work:
https://github.com/wesm/benchmarks/tree/v2-conbench-submit

R benchmark payload work:
https://github.com/wesm/arrowbench/tree/v2-conbench-payloads

There is also a live read-only evaluator running here:

https://conbench-v2.arrow-dev.org

Temporary docs are published here:

https://wesm.github.io/conbench-tmp/

The goal of this work is to provide a concrete migration path for
Conbench v2 while preserving the existing production database schema.
The evaluator is intended for review and experimentation only; it
should not modify the production database.

Please take a look at the app, docs, and workflow changes. The main
things to review are whether the new UI is useful for Arrow
maintainers, whether the benchmark reporting path is understandable,
and whether the migration approach looks practical for the existing
Arrow benchmarking workflows.

On Tue, Jun 23, 2026 at 5:34 PM Wes McKinney <we...@gmail.com> wrote:
>
> If you can give me access to the servers and cloud resources in question (ie the machines that currently run the benchmarks), I can implement the necessary code adaptations and test things working end to end, and stand up a parallel deployment of the application against RDS to enable better evaluation of the UI ergonomics. Perhaps we can coordinate offline and come back to the community with a report once the implementation is closer to being able to “throw the switch”.
>
> On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:
>>
>> Current architecture is not optimal or modern, but its maintenance cost
>> is well known and currently manageable. But I'm happy with changes that
>> move us to a more maintainable state and am willing to assist with the
>> transition.
>>
>> On a personal note, I'm hesitant to sign up for maintaining a deployment
>> of software that's not built yet.
>> So I'm just curious about who "owns" Arrow's conbench deployment
>> if I cannot commit to it.
>>
>> Rok
>>
>> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> > I think the objective is to create a more modern foundation while also
>> > improving performance. I think this looks like:
>> >
>> > * faster, easier to develop and deploy backup (use Go or Rust to create
>> > static binaries: Go is good for backend services like this)
>> > * use modern web technologies versus generating pages with Jinja templates
>> > * Use things like DuckDB and Parquet to scale result storage while
>> > improving performance
>> > * Add many more UI features to make the results most useful to maintainers
>> >
>> > Like I said, I’m happy to fulfill feature requests and contribute
>> > development with agents, if it isn’t interesting I’m also fine to go my own
>> > way.
>> >
>> > I think Buildkite is fine for job scheduling and management for now, I
>> > don’t think this system currently wants to own a task queue / durable
>> > execution state for workers, though it could grow this capability in the
>> > future (workers would have to poll the server for jobs to take).
>> >
>> > On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org> wrote:
>> >
>> > >
>> > > Hi,
>> > >
>> > > I think the main question here is: what are we trying to do?
>> > >
>> > > Currently, the main operational issue with conbench is the slowness of
>> > > the web UI, due to the large database size and that it's not normalized
>> > > (some queries take much longer than they should).
>> > >
>> > > I can't speak about the maintenance / reliability aspects, though.
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> > >
>> > > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
>> > > > hi all,
>> > > >
>> > > > I saw that conbench.ursa.dev has been down and I had a need to set up
>> > > > some continuous project benchmarks, and was interested in doing
>> > > > development on Conbench (well, having my agents do development on
>> > > > Conbench), and was interested in the following:
>> > > >
>> > > > 1) is there interest in migrating the historical Arrow conbench data
>> > > > to a new server, has that been preserved somewhere? I'll probably
>> > > > rewrite the conbench backend in Go and give it a client CLI for
>> > > > submitting new data or querying old data.
>> > > >
>> > > > 2) are there other users of conbench (conbench/conbench) that anyone
>> > > > is aware of? I'd be done doing in-situ development in that repository
>> > > > or setting up a conbench-v2 project.
>> > > >
>> > > > No particular urgency but if anyone has opinions let me know!
>> > > >
>> > > > thanks,
>> > > > Wes
>> > >
>> > >
>> >

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

If you can give me access to the servers and cloud resources in question
(ie the machines that currently run the benchmarks), I can implement the
necessary code adaptations and test things working end to end, and stand up
a parallel deployment of the application against RDS to enable better
evaluation of the UI ergonomics. Perhaps we can coordinate offline and come
back to the community with a report once the implementation is closer to
being able to “throw the switch”.

On Tue, Jun 23, 2026 at 17:02 Rok Mihevc <ro...@gmail.com> wrote:

> Current architecture is not optimal or modern, but its maintenance cost
> is well known and currently manageable. But I'm happy with changes that
> move us to a more maintainable state and am willing to assist with the
> transition.
>
> On a personal note, I'm hesitant to sign up for maintaining a deployment
> of software that's not built yet.
> So I'm just curious about who "owns" Arrow's conbench deployment
> if I cannot commit to it.
>
> Rok
>
> On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I think the objective is to create a more modern foundation while also
> > improving performance. I think this looks like:
> >
> > * faster, easier to develop and deploy backup (use Go or Rust to create
> > static binaries: Go is good for backend services like this)
> > * use modern web technologies versus generating pages with Jinja
> templates
> > * Use things like DuckDB and Parquet to scale result storage while
> > improving performance
> > * Add many more UI features to make the results most useful to
> maintainers
> >
> > Like I said, I’m happy to fulfill feature requests and contribute
> > development with agents, if it isn’t interesting I’m also fine to go my
> own
> > way.
> >
> > I think Buildkite is fine for job scheduling and management for now, I
> > don’t think this system currently wants to own a task queue / durable
> > execution state for workers, though it could grow this capability in the
> > future (workers would have to poll the server for jobs to take).
> >
> > On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Hi,
> > >
> > > I think the main question here is: what are we trying to do?
> > >
> > > Currently, the main operational issue with conbench is the slowness of
> > > the web UI, due to the large database size and that it's not normalized
> > > (some queries take much longer than they should).
> > >
> > > I can't speak about the maintenance / reliability aspects, though.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > > > hi all,
> > > >
> > > > I saw that conbench.ursa.dev has been down and I had a need to set
> up
> > > > some continuous project benchmarks, and was interested in doing
> > > > development on Conbench (well, having my agents do development on
> > > > Conbench), and was interested in the following:
> > > >
> > > > 1) is there interest in migrating the historical Arrow conbench data
> > > > to a new server, has that been preserved somewhere? I'll probably
> > > > rewrite the conbench backend in Go and give it a client CLI for
> > > > submitting new data or querying old data.
> > > >
> > > > 2) are there other users of conbench (conbench/conbench) that anyone
> > > > is aware of? I'd be done doing in-situ development in that repository
> > > > or setting up a conbench-v2 project.
> > > >
> > > > No particular urgency but if anyone has opinions let me know!
> > > >
> > > > thanks,
> > > > Wes
> > >
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Current architecture is not optimal or modern, but its maintenance cost
is well known and currently manageable. But I'm happy with changes that
move us to a more maintainable state and am willing to assist with the
transition.

On a personal note, I'm hesitant to sign up for maintaining a deployment
of software that's not built yet.
So I'm just curious about who "owns" Arrow's conbench deployment
if I cannot commit to it.

Rok

On Tue, Jun 23, 2026 at 4:40 PM Wes McKinney <we...@gmail.com> wrote:

> I think the objective is to create a more modern foundation while also
> improving performance. I think this looks like:
>
> * faster, easier to develop and deploy backup (use Go or Rust to create
> static binaries: Go is good for backend services like this)
> * use modern web technologies versus generating pages with Jinja templates
> * Use things like DuckDB and Parquet to scale result storage while
> improving performance
> * Add many more UI features to make the results most useful to maintainers
>
> Like I said, I’m happy to fulfill feature requests and contribute
> development with agents, if it isn’t interesting I’m also fine to go my own
> way.
>
> I think Buildkite is fine for job scheduling and management for now, I
> don’t think this system currently wants to own a task queue / durable
> execution state for workers, though it could grow this capability in the
> future (workers would have to poll the server for jobs to take).
>
> On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hi,
> >
> > I think the main question here is: what are we trying to do?
> >
> > Currently, the main operational issue with conbench is the slowness of
> > the web UI, due to the large database size and that it's not normalized
> > (some queries take much longer than they should).
> >
> > I can't speak about the maintenance / reliability aspects, though.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > > hi all,
> > >
> > > I saw that conbench.ursa.dev has been down and I had a need to set up
> > > some continuous project benchmarks, and was interested in doing
> > > development on Conbench (well, having my agents do development on
> > > Conbench), and was interested in the following:
> > >
> > > 1) is there interest in migrating the historical Arrow conbench data
> > > to a new server, has that been preserved somewhere? I'll probably
> > > rewrite the conbench backend in Go and give it a client CLI for
> > > submitting new data or querying old data.
> > >
> > > 2) are there other users of conbench (conbench/conbench) that anyone
> > > is aware of? I'd be done doing in-situ development in that repository
> > > or setting up a conbench-v2 project.
> > >
> > > No particular urgency but if anyone has opinions let me know!
> > >
> > > thanks,
> > > Wes
> >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

I think the objective is to create a more modern foundation while also
improving performance. I think this looks like:

* faster, easier to develop and deploy backup (use Go or Rust to create
static binaries: Go is good for backend services like this)
* use modern web technologies versus generating pages with Jinja templates
* Use things like DuckDB and Parquet to scale result storage while
improving performance
* Add many more UI features to make the results most useful to maintainers

Like I said, I’m happy to fulfill feature requests and contribute
development with agents, if it isn’t interesting I’m also fine to go my own
way.

I think Buildkite is fine for job scheduling and management for now, I
don’t think this system currently wants to own a task queue / durable
execution state for workers, though it could grow this capability in the
future (workers would have to poll the server for jobs to take).

On Tue, Jun 23, 2026 at 09:07 Antoine Pitrou <an...@python.org> wrote:

>
> Hi,
>
> I think the main question here is: what are we trying to do?
>
> Currently, the main operational issue with conbench is the slowness of
> the web UI, due to the large database size and that it's not normalized
> (some queries take much longer than they should).
>
> I can't speak about the maintenance / reliability aspects, though.
>
> Regards
>
> Antoine.
>
>
> Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> > hi all,
> >
> > I saw that conbench.ursa.dev has been down and I had a need to set up
> > some continuous project benchmarks, and was interested in doing
> > development on Conbench (well, having my agents do development on
> > Conbench), and was interested in the following:
> >
> > 1) is there interest in migrating the historical Arrow conbench data
> > to a new server, has that been preserved somewhere? I'll probably
> > rewrite the conbench backend in Go and give it a client CLI for
> > submitting new data or querying old data.
> >
> > 2) are there other users of conbench (conbench/conbench) that anyone
> > is aware of? I'd be done doing in-situ development in that repository
> > or setting up a conbench-v2 project.
> >
> > No particular urgency but if anyone has opinions let me know!
> >
> > thanks,
> > Wes
>
>

Re: Status of Arrow conbench data + conbench OSS project

[Antoine Pitrou <an...@python.org>]

Hi,

I think the main question here is: what are we trying to do?

Currently, the main operational issue with conbench is the slowness of 
the web UI, due to the large database size and that it's not normalized 
(some queries take much longer than they should).

I can't speak about the maintenance / reliability aspects, though.

Regards

Antoine.


Le 04/06/2026 à 16:19, Wes McKinney a écrit :
> hi all,
> 
> I saw that conbench.ursa.dev has been down and I had a need to set up
> some continuous project benchmarks, and was interested in doing
> development on Conbench (well, having my agents do development on
> Conbench), and was interested in the following:
> 
> 1) is there interest in migrating the historical Arrow conbench data
> to a new server, has that been preserved somewhere? I'll probably
> rewrite the conbench backend in Go and give it a client CLI for
> submitting new data or querying old data.
> 
> 2) are there other users of conbench (conbench/conbench) that anyone
> is aware of? I'd be done doing in-situ development in that repository
> or setting up a conbench-v2 project.
> 
> No particular urgency but if anyone has opinions let me know!
> 
> thanks,
> Wes

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

> > I see buildkite scheduling is removed
>
> I think this is mostly accidental, and can be easily restored

Removing buildkite is likely a good development in this case.
How is scheduling done then?


> Can you show me where the Arrow conbench suite lives so I can test /
> harden a single CI benchmark run / insert? That sounds like the next
> step, to validate things end to end and identify exactly what changes
> would be needed in the infrastructure.

Benchmarks are here https://github.com/arctosalliance/benchmarks
and here https://github.com/arctosalliance/arrowbench.


> I think the migration path would look like:
>
> 1) Start the new server in read-only mode in parallel against RDS, get
> comfortable with new UI and fill any accidental feature gaps
> 2) If step 1 seems okay and benchmark insertion runs work seamlessly
> against the new API endpoint, switch over
> 3) Explore alternative storage architectures, e.g. incrementally
> migrate historical data to Parquet while keeping recent data (e.g. the
> last 30 days worth of benchmark runs) in Postgres

I'd propose something like:
1. splice new go cli tool into the current runner so data goes to RDS and
S3.
2. stand up the new application with alerting and get comfortable with it
3. replace runner with the go cli one and decommision postgres and old app


> > On RDS we are seeing a performance bottleneck due to large tables
> > and suboptimal queries, so local postgres performance might not indicate
> > the actual RDS postgres performance.
>
> On this, since many of the Postgres queries were rewritten by gpt-5.5
> from first principles during the reimplementation process, it's
> possible that some things got faster by accident. Go vs. Python may
> also help. If there are particular workflows that are currently slow
> that I can focus on I can take a closer look.

What a time to be alive. See https://github.com/conbench/conbench/pull/1614
for the query in question.


> Note that the Go server binary can be easily wrapped in a Python wheel
> and deployed on PyPI also (that's how `pip install agentsview` works —
> wrapped Go binary)

Could be useful, not sure we need it if go cli tool is self-sufficient?


> I have a number of projects whose performance I need to continuously
> monitor, so I'd be willing to develop and maintain this new project
> with AI assistance — building this new implementation required
> probably 8 hours worth of my attention and the rest of the work was
> conducted through spec-driven development (with roborev for quality
> assurance), about $2700 worth of tokens (which all came out of a Codex
> subscription).

Great to hear about the project maintenance! Does that include deployment
maintenance? I'd be happy to step back from maintaining the current
deployment
in that case.

Thinking out loud - would it make sense to start benchmarking on one of
your new
projects first and work out the issues there before porting arrow? That
would
give maximum speed and flexiblity. Arrow's benchmark is also quite
cumbersome
and wouldn't be fun to prototype with - we currently burn ~1500 USD/moth on
AWS
runners.


Rok

On Mon, Jun 22, 2026 at 10:40 PM Wes McKinney <we...@gmail.com> wrote:

> > I see buildkite scheduling is removed
>
> I think this is mostly accidental, and can be easily restored
>
> Can you show me where the Arrow conbench suite lives so I can test /
> harden a single CI benchmark run / insert? That sounds like the next
> step, to validate things end to end and identify exactly what changes
> would be needed in the infrastructure.
>
> I think the migration path would look like:
>
> 1) Start the new server in read-only mode in parallel against RDS, get
> comfortable with new UI and fill any accidental feature gaps
> 2) If step 1 seems okay and benchmark insertion runs work seamlessly
> against the new API endpoint, switch over
> 3) Explore alternative storage architectures, e.g. incrementally
> migrate historical data to Parquet while keeping recent data (e.g. the
> last 30 days worth of benchmark runs) in Postgres
>
> > On RDS we are seeing a performance bottleneck due to large tables
> and suboptimal queries, so local postgres performance might not indicate
> the actual RDS postgres performance.
>
> On this, since many of the Postgres queries were rewritten by gpt-5.5
> from first principles during the reimplementation process, it's
> possible that some things got faster by accident. Go vs. Python may
> also help. If there are particular workflows that are currently slow
> that I can focus on I can take a closer look.
>
> Note that the Go server binary can be easily wrapped in a Python wheel
> and deployed on PyPI also (that's how `pip install agentsview` works —
> wrapped Go binary)
>
> I have a number of projects whose performance I need to continuously
> monitor, so I'd be willing to develop and maintain this new project
> with AI assistance — building this new implementation required
> probably 8 hours worth of my attention and the rest of the work was
> conducted through spec-driven development (with roborev for quality
> assurance), about $2700 worth of tokens (which all came out of a Codex
> subscription).
>
> Thanks,
> Wes
>
> On Mon, Jun 22, 2026 at 3:22 PM Rok Mihevc <ro...@gmail.com> wrote:
> >
> > Hi Wes,
> >
> > > The general idea is that there is a new backend in Go and a single-page
> > Typescript frontend application.
> >
> > Single tool approach seems preferable over the current implementation for
> > simplicity etc. I see buildkite scheduling is removed; does the core app
> > now handle scheduling? I assume the DB schema and core alghoritms are
> kept.
> > This moves in the right direction.
> >
> > > My general sense is that at least for result data storage, using e.g.
> > DuckDB + Parquet files on object storage might be a viable alternative to
> > Postgres to tinker with, but the performance of this application against
> a
> > local Postgres database seems acceptable.
> >
> > I'm curious about postgres performance - are you using current queries or
> > new ones? On RDS we are seeing a performance bottleneck due to large
> tables
> > and suboptimal queries, so local postgres performance might not indicate
> > the actual RDS postgres performance.
> > That said I believe parquet files on object storage would be preferable
> > anyway. Ideally, the conbench result viewer should (IMO) only be a view
> > layer over precomputed data generated at benchmark or compaction time.
> >
> > > I am not expecting this PR to be accepted necessarily but mostly using
> it
> > to see if there is interest in going down this path! I am also okay to
> > create a new git repository and do green field development, or let it
> live
> > in a branch that can get developed an experimented with for a period of
> > time (e.g. it seems like part of this project would be proving out
> getting
> > the Arrow benchmark runners to upload their results using the new HTTP
> API
> > or Go CLI).
> >
> > My question is mainly - Does this proposal move Arrow's conbench instance
> > to a more maintainable state?
> >
> > Having migrated and maintained Arrow's current conbench deployment, I
> would
> > love to see it simplified for easier maintenance and I'm happy to help
> move
> > our deployment to a better state.
> > That said do you intend to maintain conbench going forward? Who do you
> see
> > maintaining Arrow's conbench deployment?
> >
> > Rok
> >
> > On Sun, Jun 21, 2026 at 9:28 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > I had an opportunity to grind on this for a couple weeks with spare
> > > coding agent cycles and produced this pull request, which I've stood
> > > up successfully against the clone of the prod Postgres database that
> > > Rok shared with me. The general idea is that there is a new backend in
> > > Go and a single-page Typescript frontend application.
> > >
> > > My general sense is that at least for result data storage, using e.g.
> > > DuckDB + Parquet files on object storage might be a viable alternative
> > > to Postgres to tinker with, but the performance of this application
> > > against a local Postgres database seems acceptable.
> > >
> > > https://github.com/conbench/conbench/pull/1619
> > >
> > > I stood up a documentation site here:
> > >
> > > https://wesm.github.io/conbench-tmp/
> > >
> > > Example screenshots:
> > >
> > > https://wesm.github.io/conbench-tmp/dashboard-screenshots/
> > >
> > > I am not expecting this PR to be accepted necessarily but mostly using
> > > it to see if there is interest in going down this path! I am also okay
> > > to create a new git repository and do green field development, or let
> > > it live in a branch that can get developed an experimented with for a
> > > period of time (e.g. it seems like part of this project would be
> > > proving out getting the Arrow benchmark runners to upload their
> > > results using the new HTTP API or Go CLI).
> > >
> > > Thanks,
> > > Wes
> > >
> > > On Thu, Jun 4, 2026 at 9:26 PM Jonathan Keane <jk...@gmail.com>
> wrote:
> > > >
> > > > Thanks for this. I also don’t know of anyone else using conbench
> > > currently, but I am happy to hear that you are interested in (having
> your
> > > agents) work on it.
> > > >
> > > > -Jon
> > > >
> > > > > On Jun 4, 2026, at 16:45, Rok Mihevc <ro...@gmail.com> wrote:
> > > > >
> > > > > Will do! Looking forward to hearing what you're looking to achieve.
> > > > >
> > > > > Perhaps we can use the attention here (if we got any) to discuss
> what
> > > Arrow
> > > > > needs/wants in terms of benchmarking functionality.
> > > > >
> > > > >
> > > > > On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Okay sounds good! I’ll start a v2 branch and do some work on it.
> Can
> > > you
> > > > >> contact me offline and let me know how I can get a copy of the
> prod
> > > > >> database to help with development and test migrations?
> > > > >>
> > > > >> Thanks
> > > > >> Wes
> > > > >>
> > > > >> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com>
> wrote:
> > > > >>
> > > > >>> As per my discussion with the Velox team in March, they weren't
> using
> > > > >>> Conbench at the time and had no plans to migrate back.
> > > > >>>
> > > > >>> I never reported on the Conbench/Voltron migration to the mailing
> > > list,
> > > > >> so
> > > > >>> this is a good opportunity to do so:
> > > > >>> - In November, it became clear Voltron Data would shut down its
> CI
> > > and
> > > > >>> benchmarking infrastructure
> > > > >>> - Using the donated AWS credits [1] and with the help of Mike
> Wendt I
> > > > >> moved
> > > > >>> conbench infra to the new AWS account and backfilled benchmarks
> to
> > > cover
> > > > >>> the missing commits
> > > > >>> - Raul and I moved CUDA runners to the new AWS account
> > > > >>> - Moved doc-preview, Parquet benchmarking/testing datasets, the
> > > nightly
> > > > >>> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> > > > >>> - Conbench gradually stabilized, but benchmarks occasionally fail
> > > due to
> > > > >>> dependency issues. We don't really track these, but Nic and I
> > > typically
> > > > >> fix
> > > > >>> them.
> > > > >>> - The database design still makes the web views slow, and we
> > > discussed
> > > > >>> about dropping some history to speed them up
> > > > >>> - AWS benchmark runner-minutes consume nearly the entire donated
> AWS
> > > > >> quota.
> > > > >>> Discussion on how to optimize this and what exactly we need from
> > > > >> benchmarks
> > > > >>> as a project occasionally comes up on Zulip, and I expect that
> once
> > > it
> > > > >>> crystallizes we'll bring it to the ML as well.
> > > > >>> - Things are now stable enough that we can discuss redesigning
> > > toward a
> > > > >>> more stable and sustainable setup
> > > > >>>
> > > > >>> Best,
> > > > >>> Rok
> > > > >>>
> > > > >>> [1]
> https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
> > > > >>>
> > > > >>> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <
> assignuser@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> +1 improving conbench and the orchestration would free up aws
> > > credits
> > > > >> and
> > > > >>>> more importantly the time taken to fix outrages!
> > > > >>>>
> > > > >>>> Velox used to use conbench but through a VD provided instance. I
> > > > >> haven't
> > > > >>>> seen any movement to set up an independent instance.
> > > > >>>>
> > > > >>>>
> > > > >>>> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026,
> > > 16:38:
> > > > >>>>
> > > > >>>>> Hi Wes,
> > > > >>>>>
> > > > >>>>> Conbench is now at conbench.arrow-dev.org. I forked all repos
> to
> > > > >>>>> arctosalliance github org in case VD deleted them on the way
> out.
> > > > >>>>> Historical RDS db is preserved in the AWS account we migrated
> > > Arrow's
> > > > >>> CI
> > > > >>>>> infra to. It's supported by donated AWS credits.
> > > > >>>>> For orchestration buildkite.com/apache-arrow is still used
> and I
> > > > >> even
> > > > >>>> got
> > > > >>>>> the old M1 mac mini from Mike. It's now in my rack running
> > > benchmarks
> > > > >>>>> again.
> > > > >>>>>
> > > > >>>>> That said current design of the orchestration layer, api
> server and
> > > > >> db
> > > > >>> is
> > > > >>>>> somewhat costly and brittle. Downtime and adhoc fixes are
> common
> > > and
> > > > >> I
> > > > >>>> was
> > > > >>>>> thinking about refactoring it somewhat to make it more
> > > maintainable.
> > > > >>>>>
> > > > >>>>> 1) If things become more maintainable and long term stable
> that'd
> > > be
> > > > >>>> great.
> > > > >>>>> I'm happy to collaborate on this.
> > > > >>>>> 2) I'm only aware of arrow and arrow-go currently using it.
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Rok
> > > > >>>>>
> > > > >>>>> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>>> hi all,
> > > > >>>>>>
> > > > >>>>>> I saw that conbench.ursa.dev has been down and I had a need
> to
> > > set
> > > > >>> up
> > > > >>>>>> some continuous project benchmarks, and was interested in
> doing
> > > > >>>>>> development on Conbench (well, having my agents do
> development on
> > > > >>>>>> Conbench), and was interested in the following:
> > > > >>>>>>
> > > > >>>>>> 1) is there interest in migrating the historical Arrow
> conbench
> > > > >> data
> > > > >>>>>> to a new server, has that been preserved somewhere? I'll
> probably
> > > > >>>>>> rewrite the conbench backend in Go and give it a client CLI
> for
> > > > >>>>>> submitting new data or querying old data.
> > > > >>>>>>
> > > > >>>>>> 2) are there other users of conbench (conbench/conbench) that
> > > > >> anyone
> > > > >>>>>> is aware of? I'd be done doing in-situ development in that
> > > > >> repository
> > > > >>>>>> or setting up a conbench-v2 project.
> > > > >>>>>>
> > > > >>>>>> No particular urgency but if anyone has opinions let me know!
> > > > >>>>>>
> > > > >>>>>> thanks,
> > > > >>>>>> Wes
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

> I see buildkite scheduling is removed

I think this is mostly accidental, and can be easily restored

Can you show me where the Arrow conbench suite lives so I can test /
harden a single CI benchmark run / insert? That sounds like the next
step, to validate things end to end and identify exactly what changes
would be needed in the infrastructure.

I think the migration path would look like:

1) Start the new server in read-only mode in parallel against RDS, get
comfortable with new UI and fill any accidental feature gaps
2) If step 1 seems okay and benchmark insertion runs work seamlessly
against the new API endpoint, switch over
3) Explore alternative storage architectures, e.g. incrementally
migrate historical data to Parquet while keeping recent data (e.g. the
last 30 days worth of benchmark runs) in Postgres

> On RDS we are seeing a performance bottleneck due to large tables
and suboptimal queries, so local postgres performance might not indicate
the actual RDS postgres performance.

On this, since many of the Postgres queries were rewritten by gpt-5.5
from first principles during the reimplementation process, it's
possible that some things got faster by accident. Go vs. Python may
also help. If there are particular workflows that are currently slow
that I can focus on I can take a closer look.

Note that the Go server binary can be easily wrapped in a Python wheel
and deployed on PyPI also (that's how `pip install agentsview` works —
wrapped Go binary)

I have a number of projects whose performance I need to continuously
monitor, so I'd be willing to develop and maintain this new project
with AI assistance — building this new implementation required
probably 8 hours worth of my attention and the rest of the work was
conducted through spec-driven development (with roborev for quality
assurance), about $2700 worth of tokens (which all came out of a Codex
subscription).

Thanks,
Wes

On Mon, Jun 22, 2026 at 3:22 PM Rok Mihevc <ro...@gmail.com> wrote:
>
> Hi Wes,
>
> > The general idea is that there is a new backend in Go and a single-page
> Typescript frontend application.
>
> Single tool approach seems preferable over the current implementation for
> simplicity etc. I see buildkite scheduling is removed; does the core app
> now handle scheduling? I assume the DB schema and core alghoritms are kept.
> This moves in the right direction.
>
> > My general sense is that at least for result data storage, using e.g.
> DuckDB + Parquet files on object storage might be a viable alternative to
> Postgres to tinker with, but the performance of this application against a
> local Postgres database seems acceptable.
>
> I'm curious about postgres performance - are you using current queries or
> new ones? On RDS we are seeing a performance bottleneck due to large tables
> and suboptimal queries, so local postgres performance might not indicate
> the actual RDS postgres performance.
> That said I believe parquet files on object storage would be preferable
> anyway. Ideally, the conbench result viewer should (IMO) only be a view
> layer over precomputed data generated at benchmark or compaction time.
>
> > I am not expecting this PR to be accepted necessarily but mostly using it
> to see if there is interest in going down this path! I am also okay to
> create a new git repository and do green field development, or let it live
> in a branch that can get developed an experimented with for a period of
> time (e.g. it seems like part of this project would be proving out getting
> the Arrow benchmark runners to upload their results using the new HTTP API
> or Go CLI).
>
> My question is mainly - Does this proposal move Arrow's conbench instance
> to a more maintainable state?
>
> Having migrated and maintained Arrow's current conbench deployment, I would
> love to see it simplified for easier maintenance and I'm happy to help move
> our deployment to a better state.
> That said do you intend to maintain conbench going forward? Who do you see
> maintaining Arrow's conbench deployment?
>
> Rok
>
> On Sun, Jun 21, 2026 at 9:28 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I had an opportunity to grind on this for a couple weeks with spare
> > coding agent cycles and produced this pull request, which I've stood
> > up successfully against the clone of the prod Postgres database that
> > Rok shared with me. The general idea is that there is a new backend in
> > Go and a single-page Typescript frontend application.
> >
> > My general sense is that at least for result data storage, using e.g.
> > DuckDB + Parquet files on object storage might be a viable alternative
> > to Postgres to tinker with, but the performance of this application
> > against a local Postgres database seems acceptable.
> >
> > https://github.com/conbench/conbench/pull/1619
> >
> > I stood up a documentation site here:
> >
> > https://wesm.github.io/conbench-tmp/
> >
> > Example screenshots:
> >
> > https://wesm.github.io/conbench-tmp/dashboard-screenshots/
> >
> > I am not expecting this PR to be accepted necessarily but mostly using
> > it to see if there is interest in going down this path! I am also okay
> > to create a new git repository and do green field development, or let
> > it live in a branch that can get developed an experimented with for a
> > period of time (e.g. it seems like part of this project would be
> > proving out getting the Arrow benchmark runners to upload their
> > results using the new HTTP API or Go CLI).
> >
> > Thanks,
> > Wes
> >
> > On Thu, Jun 4, 2026 at 9:26 PM Jonathan Keane <jk...@gmail.com> wrote:
> > >
> > > Thanks for this. I also don’t know of anyone else using conbench
> > currently, but I am happy to hear that you are interested in (having your
> > agents) work on it.
> > >
> > > -Jon
> > >
> > > > On Jun 4, 2026, at 16:45, Rok Mihevc <ro...@gmail.com> wrote:
> > > >
> > > > Will do! Looking forward to hearing what you're looking to achieve.
> > > >
> > > > Perhaps we can use the attention here (if we got any) to discuss what
> > Arrow
> > > > needs/wants in terms of benchmarking functionality.
> > > >
> > > >
> > > > On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > >> Okay sounds good! I’ll start a v2 branch and do some work on it. Can
> > you
> > > >> contact me offline and let me know how I can get a copy of the prod
> > > >> database to help with development and test migrations?
> > > >>
> > > >> Thanks
> > > >> Wes
> > > >>
> > > >> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:
> > > >>
> > > >>> As per my discussion with the Velox team in March, they weren't using
> > > >>> Conbench at the time and had no plans to migrate back.
> > > >>>
> > > >>> I never reported on the Conbench/Voltron migration to the mailing
> > list,
> > > >> so
> > > >>> this is a good opportunity to do so:
> > > >>> - In November, it became clear Voltron Data would shut down its CI
> > and
> > > >>> benchmarking infrastructure
> > > >>> - Using the donated AWS credits [1] and with the help of Mike Wendt I
> > > >> moved
> > > >>> conbench infra to the new AWS account and backfilled benchmarks to
> > cover
> > > >>> the missing commits
> > > >>> - Raul and I moved CUDA runners to the new AWS account
> > > >>> - Moved doc-preview, Parquet benchmarking/testing datasets, the
> > nightly
> > > >>> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> > > >>> - Conbench gradually stabilized, but benchmarks occasionally fail
> > due to
> > > >>> dependency issues. We don't really track these, but Nic and I
> > typically
> > > >> fix
> > > >>> them.
> > > >>> - The database design still makes the web views slow, and we
> > discussed
> > > >>> about dropping some history to speed them up
> > > >>> - AWS benchmark runner-minutes consume nearly the entire donated AWS
> > > >> quota.
> > > >>> Discussion on how to optimize this and what exactly we need from
> > > >> benchmarks
> > > >>> as a project occasionally comes up on Zulip, and I expect that once
> > it
> > > >>> crystallizes we'll bring it to the ML as well.
> > > >>> - Things are now stable enough that we can discuss redesigning
> > toward a
> > > >>> more stable and sustainable setup
> > > >>>
> > > >>> Best,
> > > >>> Rok
> > > >>>
> > > >>> [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
> > > >>>
> > > >>> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 improving conbench and the orchestration would free up aws
> > credits
> > > >> and
> > > >>>> more importantly the time taken to fix outrages!
> > > >>>>
> > > >>>> Velox used to use conbench but through a VD provided instance. I
> > > >> haven't
> > > >>>> seen any movement to set up an independent instance.
> > > >>>>
> > > >>>>
> > > >>>> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026,
> > 16:38:
> > > >>>>
> > > >>>>> Hi Wes,
> > > >>>>>
> > > >>>>> Conbench is now at conbench.arrow-dev.org. I forked all repos to
> > > >>>>> arctosalliance github org in case VD deleted them on the way out.
> > > >>>>> Historical RDS db is preserved in the AWS account we migrated
> > Arrow's
> > > >>> CI
> > > >>>>> infra to. It's supported by donated AWS credits.
> > > >>>>> For orchestration buildkite.com/apache-arrow is still used and I
> > > >> even
> > > >>>> got
> > > >>>>> the old M1 mac mini from Mike. It's now in my rack running
> > benchmarks
> > > >>>>> again.
> > > >>>>>
> > > >>>>> That said current design of the orchestration layer, api server and
> > > >> db
> > > >>> is
> > > >>>>> somewhat costly and brittle. Downtime and adhoc fixes are common
> > and
> > > >> I
> > > >>>> was
> > > >>>>> thinking about refactoring it somewhat to make it more
> > maintainable.
> > > >>>>>
> > > >>>>> 1) If things become more maintainable and long term stable that'd
> > be
> > > >>>> great.
> > > >>>>> I'm happy to collaborate on this.
> > > >>>>> 2) I'm only aware of arrow and arrow-go currently using it.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Rok
> > > >>>>>
> > > >>>>> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> hi all,
> > > >>>>>>
> > > >>>>>> I saw that conbench.ursa.dev has been down and I had a need to
> > set
> > > >>> up
> > > >>>>>> some continuous project benchmarks, and was interested in doing
> > > >>>>>> development on Conbench (well, having my agents do development on
> > > >>>>>> Conbench), and was interested in the following:
> > > >>>>>>
> > > >>>>>> 1) is there interest in migrating the historical Arrow conbench
> > > >> data
> > > >>>>>> to a new server, has that been preserved somewhere? I'll probably
> > > >>>>>> rewrite the conbench backend in Go and give it a client CLI for
> > > >>>>>> submitting new data or querying old data.
> > > >>>>>>
> > > >>>>>> 2) are there other users of conbench (conbench/conbench) that
> > > >> anyone
> > > >>>>>> is aware of? I'd be done doing in-situ development in that
> > > >> repository
> > > >>>>>> or setting up a conbench-v2 project.
> > > >>>>>>
> > > >>>>>> No particular urgency but if anyone has opinions let me know!
> > > >>>>>>
> > > >>>>>> thanks,
> > > >>>>>> Wes
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Hi Wes,

> The general idea is that there is a new backend in Go and a single-page
Typescript frontend application.

Single tool approach seems preferable over the current implementation for
simplicity etc. I see buildkite scheduling is removed; does the core app
now handle scheduling? I assume the DB schema and core alghoritms are kept.
This moves in the right direction.

> My general sense is that at least for result data storage, using e.g.
DuckDB + Parquet files on object storage might be a viable alternative to
Postgres to tinker with, but the performance of this application against a
local Postgres database seems acceptable.

I'm curious about postgres performance - are you using current queries or
new ones? On RDS we are seeing a performance bottleneck due to large tables
and suboptimal queries, so local postgres performance might not indicate
the actual RDS postgres performance.
That said I believe parquet files on object storage would be preferable
anyway. Ideally, the conbench result viewer should (IMO) only be a view
layer over precomputed data generated at benchmark or compaction time.

> I am not expecting this PR to be accepted necessarily but mostly using it
to see if there is interest in going down this path! I am also okay to
create a new git repository and do green field development, or let it live
in a branch that can get developed an experimented with for a period of
time (e.g. it seems like part of this project would be proving out getting
the Arrow benchmark runners to upload their results using the new HTTP API
or Go CLI).

My question is mainly - Does this proposal move Arrow's conbench instance
to a more maintainable state?

Having migrated and maintained Arrow's current conbench deployment, I would
love to see it simplified for easier maintenance and I'm happy to help move
our deployment to a better state.
That said do you intend to maintain conbench going forward? Who do you see
maintaining Arrow's conbench deployment?

Rok

On Sun, Jun 21, 2026 at 9:28 PM Wes McKinney <we...@gmail.com> wrote:

> I had an opportunity to grind on this for a couple weeks with spare
> coding agent cycles and produced this pull request, which I've stood
> up successfully against the clone of the prod Postgres database that
> Rok shared with me. The general idea is that there is a new backend in
> Go and a single-page Typescript frontend application.
>
> My general sense is that at least for result data storage, using e.g.
> DuckDB + Parquet files on object storage might be a viable alternative
> to Postgres to tinker with, but the performance of this application
> against a local Postgres database seems acceptable.
>
> https://github.com/conbench/conbench/pull/1619
>
> I stood up a documentation site here:
>
> https://wesm.github.io/conbench-tmp/
>
> Example screenshots:
>
> https://wesm.github.io/conbench-tmp/dashboard-screenshots/
>
> I am not expecting this PR to be accepted necessarily but mostly using
> it to see if there is interest in going down this path! I am also okay
> to create a new git repository and do green field development, or let
> it live in a branch that can get developed an experimented with for a
> period of time (e.g. it seems like part of this project would be
> proving out getting the Arrow benchmark runners to upload their
> results using the new HTTP API or Go CLI).
>
> Thanks,
> Wes
>
> On Thu, Jun 4, 2026 at 9:26 PM Jonathan Keane <jk...@gmail.com> wrote:
> >
> > Thanks for this. I also don’t know of anyone else using conbench
> currently, but I am happy to hear that you are interested in (having your
> agents) work on it.
> >
> > -Jon
> >
> > > On Jun 4, 2026, at 16:45, Rok Mihevc <ro...@gmail.com> wrote:
> > >
> > > Will do! Looking forward to hearing what you're looking to achieve.
> > >
> > > Perhaps we can use the attention here (if we got any) to discuss what
> Arrow
> > > needs/wants in terms of benchmarking functionality.
> > >
> > >
> > > On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > >> Okay sounds good! I’ll start a v2 branch and do some work on it. Can
> you
> > >> contact me offline and let me know how I can get a copy of the prod
> > >> database to help with development and test migrations?
> > >>
> > >> Thanks
> > >> Wes
> > >>
> > >> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:
> > >>
> > >>> As per my discussion with the Velox team in March, they weren't using
> > >>> Conbench at the time and had no plans to migrate back.
> > >>>
> > >>> I never reported on the Conbench/Voltron migration to the mailing
> list,
> > >> so
> > >>> this is a good opportunity to do so:
> > >>> - In November, it became clear Voltron Data would shut down its CI
> and
> > >>> benchmarking infrastructure
> > >>> - Using the donated AWS credits [1] and with the help of Mike Wendt I
> > >> moved
> > >>> conbench infra to the new AWS account and backfilled benchmarks to
> cover
> > >>> the missing commits
> > >>> - Raul and I moved CUDA runners to the new AWS account
> > >>> - Moved doc-preview, Parquet benchmarking/testing datasets, the
> nightly
> > >>> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> > >>> - Conbench gradually stabilized, but benchmarks occasionally fail
> due to
> > >>> dependency issues. We don't really track these, but Nic and I
> typically
> > >> fix
> > >>> them.
> > >>> - The database design still makes the web views slow, and we
> discussed
> > >>> about dropping some history to speed them up
> > >>> - AWS benchmark runner-minutes consume nearly the entire donated AWS
> > >> quota.
> > >>> Discussion on how to optimize this and what exactly we need from
> > >> benchmarks
> > >>> as a project occasionally comes up on Zulip, and I expect that once
> it
> > >>> crystallizes we'll bring it to the ML as well.
> > >>> - Things are now stable enough that we can discuss redesigning
> toward a
> > >>> more stable and sustainable setup
> > >>>
> > >>> Best,
> > >>> Rok
> > >>>
> > >>> [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
> > >>>
> > >>> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
> > >>> wrote:
> > >>>
> > >>>> +1 improving conbench and the orchestration would free up aws
> credits
> > >> and
> > >>>> more importantly the time taken to fix outrages!
> > >>>>
> > >>>> Velox used to use conbench but through a VD provided instance. I
> > >> haven't
> > >>>> seen any movement to set up an independent instance.
> > >>>>
> > >>>>
> > >>>> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026,
> 16:38:
> > >>>>
> > >>>>> Hi Wes,
> > >>>>>
> > >>>>> Conbench is now at conbench.arrow-dev.org. I forked all repos to
> > >>>>> arctosalliance github org in case VD deleted them on the way out.
> > >>>>> Historical RDS db is preserved in the AWS account we migrated
> Arrow's
> > >>> CI
> > >>>>> infra to. It's supported by donated AWS credits.
> > >>>>> For orchestration buildkite.com/apache-arrow is still used and I
> > >> even
> > >>>> got
> > >>>>> the old M1 mac mini from Mike. It's now in my rack running
> benchmarks
> > >>>>> again.
> > >>>>>
> > >>>>> That said current design of the orchestration layer, api server and
> > >> db
> > >>> is
> > >>>>> somewhat costly and brittle. Downtime and adhoc fixes are common
> and
> > >> I
> > >>>> was
> > >>>>> thinking about refactoring it somewhat to make it more
> maintainable.
> > >>>>>
> > >>>>> 1) If things become more maintainable and long term stable that'd
> be
> > >>>> great.
> > >>>>> I'm happy to collaborate on this.
> > >>>>> 2) I'm only aware of arrow and arrow-go currently using it.
> > >>>>>
> > >>>>> Best,
> > >>>>> Rok
> > >>>>>
> > >>>>> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> hi all,
> > >>>>>>
> > >>>>>> I saw that conbench.ursa.dev has been down and I had a need to
> set
> > >>> up
> > >>>>>> some continuous project benchmarks, and was interested in doing
> > >>>>>> development on Conbench (well, having my agents do development on
> > >>>>>> Conbench), and was interested in the following:
> > >>>>>>
> > >>>>>> 1) is there interest in migrating the historical Arrow conbench
> > >> data
> > >>>>>> to a new server, has that been preserved somewhere? I'll probably
> > >>>>>> rewrite the conbench backend in Go and give it a client CLI for
> > >>>>>> submitting new data or querying old data.
> > >>>>>>
> > >>>>>> 2) are there other users of conbench (conbench/conbench) that
> > >> anyone
> > >>>>>> is aware of? I'd be done doing in-situ development in that
> > >> repository
> > >>>>>> or setting up a conbench-v2 project.
> > >>>>>>
> > >>>>>> No particular urgency but if anyone has opinions let me know!
> > >>>>>>
> > >>>>>> thanks,
> > >>>>>> Wes
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

I had an opportunity to grind on this for a couple weeks with spare
coding agent cycles and produced this pull request, which I've stood
up successfully against the clone of the prod Postgres database that
Rok shared with me. The general idea is that there is a new backend in
Go and a single-page Typescript frontend application.

My general sense is that at least for result data storage, using e.g.
DuckDB + Parquet files on object storage might be a viable alternative
to Postgres to tinker with, but the performance of this application
against a local Postgres database seems acceptable.

https://github.com/conbench/conbench/pull/1619

I stood up a documentation site here:

https://wesm.github.io/conbench-tmp/

Example screenshots:

https://wesm.github.io/conbench-tmp/dashboard-screenshots/

I am not expecting this PR to be accepted necessarily but mostly using
it to see if there is interest in going down this path! I am also okay
to create a new git repository and do green field development, or let
it live in a branch that can get developed an experimented with for a
period of time (e.g. it seems like part of this project would be
proving out getting the Arrow benchmark runners to upload their
results using the new HTTP API or Go CLI).

Thanks,
Wes

On Thu, Jun 4, 2026 at 9:26 PM Jonathan Keane <jk...@gmail.com> wrote:
>
> Thanks for this. I also don’t know of anyone else using conbench currently, but I am happy to hear that you are interested in (having your agents) work on it.
>
> -Jon
>
> > On Jun 4, 2026, at 16:45, Rok Mihevc <ro...@gmail.com> wrote:
> >
> > Will do! Looking forward to hearing what you're looking to achieve.
> >
> > Perhaps we can use the attention here (if we got any) to discuss what Arrow
> > needs/wants in terms of benchmarking functionality.
> >
> >
> > On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com> wrote:
> >
> >> Okay sounds good! I’ll start a v2 branch and do some work on it. Can you
> >> contact me offline and let me know how I can get a copy of the prod
> >> database to help with development and test migrations?
> >>
> >> Thanks
> >> Wes
> >>
> >> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:
> >>
> >>> As per my discussion with the Velox team in March, they weren't using
> >>> Conbench at the time and had no plans to migrate back.
> >>>
> >>> I never reported on the Conbench/Voltron migration to the mailing list,
> >> so
> >>> this is a good opportunity to do so:
> >>> - In November, it became clear Voltron Data would shut down its CI and
> >>> benchmarking infrastructure
> >>> - Using the donated AWS credits [1] and with the help of Mike Wendt I
> >> moved
> >>> conbench infra to the new AWS account and backfilled benchmarks to cover
> >>> the missing commits
> >>> - Raul and I moved CUDA runners to the new AWS account
> >>> - Moved doc-preview, Parquet benchmarking/testing datasets, the nightly
> >>> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> >>> - Conbench gradually stabilized, but benchmarks occasionally fail due to
> >>> dependency issues. We don't really track these, but Nic and I typically
> >> fix
> >>> them.
> >>> - The database design still makes the web views slow, and we discussed
> >>> about dropping some history to speed them up
> >>> - AWS benchmark runner-minutes consume nearly the entire donated AWS
> >> quota.
> >>> Discussion on how to optimize this and what exactly we need from
> >> benchmarks
> >>> as a project occasionally comes up on Zulip, and I expect that once it
> >>> crystallizes we'll bring it to the ML as well.
> >>> - Things are now stable enough that we can discuss redesigning toward a
> >>> more stable and sustainable setup
> >>>
> >>> Best,
> >>> Rok
> >>>
> >>> [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
> >>>
> >>> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
> >>> wrote:
> >>>
> >>>> +1 improving conbench and the orchestration would free up aws credits
> >> and
> >>>> more importantly the time taken to fix outrages!
> >>>>
> >>>> Velox used to use conbench but through a VD provided instance. I
> >> haven't
> >>>> seen any movement to set up an independent instance.
> >>>>
> >>>>
> >>>> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:
> >>>>
> >>>>> Hi Wes,
> >>>>>
> >>>>> Conbench is now at conbench.arrow-dev.org. I forked all repos to
> >>>>> arctosalliance github org in case VD deleted them on the way out.
> >>>>> Historical RDS db is preserved in the AWS account we migrated Arrow's
> >>> CI
> >>>>> infra to. It's supported by donated AWS credits.
> >>>>> For orchestration buildkite.com/apache-arrow is still used and I
> >> even
> >>>> got
> >>>>> the old M1 mac mini from Mike. It's now in my rack running benchmarks
> >>>>> again.
> >>>>>
> >>>>> That said current design of the orchestration layer, api server and
> >> db
> >>> is
> >>>>> somewhat costly and brittle. Downtime and adhoc fixes are common and
> >> I
> >>>> was
> >>>>> thinking about refactoring it somewhat to make it more maintainable.
> >>>>>
> >>>>> 1) If things become more maintainable and long term stable that'd be
> >>>> great.
> >>>>> I'm happy to collaborate on this.
> >>>>> 2) I'm only aware of arrow and arrow-go currently using it.
> >>>>>
> >>>>> Best,
> >>>>> Rok
> >>>>>
> >>>>> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> hi all,
> >>>>>>
> >>>>>> I saw that conbench.ursa.dev has been down and I had a need to set
> >>> up
> >>>>>> some continuous project benchmarks, and was interested in doing
> >>>>>> development on Conbench (well, having my agents do development on
> >>>>>> Conbench), and was interested in the following:
> >>>>>>
> >>>>>> 1) is there interest in migrating the historical Arrow conbench
> >> data
> >>>>>> to a new server, has that been preserved somewhere? I'll probably
> >>>>>> rewrite the conbench backend in Go and give it a client CLI for
> >>>>>> submitting new data or querying old data.
> >>>>>>
> >>>>>> 2) are there other users of conbench (conbench/conbench) that
> >> anyone
> >>>>>> is aware of? I'd be done doing in-situ development in that
> >> repository
> >>>>>> or setting up a conbench-v2 project.
> >>>>>>
> >>>>>> No particular urgency but if anyone has opinions let me know!
> >>>>>>
> >>>>>> thanks,
> >>>>>> Wes
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Re: Status of Arrow conbench data + conbench OSS project

[Jonathan Keane <jk...@gmail.com>]

Thanks for this. I also don’t know of anyone else using conbench currently, but I am happy to hear that you are interested in (having your agents) work on it.

-Jon

> On Jun 4, 2026, at 16:45, Rok Mihevc <ro...@gmail.com> wrote:
> 
> Will do! Looking forward to hearing what you're looking to achieve.
> 
> Perhaps we can use the attention here (if we got any) to discuss what Arrow
> needs/wants in terms of benchmarking functionality.
> 
> 
> On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com> wrote:
> 
>> Okay sounds good! I’ll start a v2 branch and do some work on it. Can you
>> contact me offline and let me know how I can get a copy of the prod
>> database to help with development and test migrations?
>> 
>> Thanks
>> Wes
>> 
>> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:
>> 
>>> As per my discussion with the Velox team in March, they weren't using
>>> Conbench at the time and had no plans to migrate back.
>>> 
>>> I never reported on the Conbench/Voltron migration to the mailing list,
>> so
>>> this is a good opportunity to do so:
>>> - In November, it became clear Voltron Data would shut down its CI and
>>> benchmarking infrastructure
>>> - Using the donated AWS credits [1] and with the help of Mike Wendt I
>> moved
>>> conbench infra to the new AWS account and backfilled benchmarks to cover
>>> the missing commits
>>> - Raul and I moved CUDA runners to the new AWS account
>>> - Moved doc-preview, Parquet benchmarking/testing datasets, the nightly
>>> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
>>> - Conbench gradually stabilized, but benchmarks occasionally fail due to
>>> dependency issues. We don't really track these, but Nic and I typically
>> fix
>>> them.
>>> - The database design still makes the web views slow, and we discussed
>>> about dropping some history to speed them up
>>> - AWS benchmark runner-minutes consume nearly the entire donated AWS
>> quota.
>>> Discussion on how to optimize this and what exactly we need from
>> benchmarks
>>> as a project occasionally comes up on Zulip, and I expect that once it
>>> crystallizes we'll bring it to the ML as well.
>>> - Things are now stable enough that we can discuss redesigning toward a
>>> more stable and sustainable setup
>>> 
>>> Best,
>>> Rok
>>> 
>>> [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
>>> 
>>> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
>>> wrote:
>>> 
>>>> +1 improving conbench and the orchestration would free up aws credits
>> and
>>>> more importantly the time taken to fix outrages!
>>>> 
>>>> Velox used to use conbench but through a VD provided instance. I
>> haven't
>>>> seen any movement to set up an independent instance.
>>>> 
>>>> 
>>>> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:
>>>> 
>>>>> Hi Wes,
>>>>> 
>>>>> Conbench is now at conbench.arrow-dev.org. I forked all repos to
>>>>> arctosalliance github org in case VD deleted them on the way out.
>>>>> Historical RDS db is preserved in the AWS account we migrated Arrow's
>>> CI
>>>>> infra to. It's supported by donated AWS credits.
>>>>> For orchestration buildkite.com/apache-arrow is still used and I
>> even
>>>> got
>>>>> the old M1 mac mini from Mike. It's now in my rack running benchmarks
>>>>> again.
>>>>> 
>>>>> That said current design of the orchestration layer, api server and
>> db
>>> is
>>>>> somewhat costly and brittle. Downtime and adhoc fixes are common and
>> I
>>>> was
>>>>> thinking about refactoring it somewhat to make it more maintainable.
>>>>> 
>>>>> 1) If things become more maintainable and long term stable that'd be
>>>> great.
>>>>> I'm happy to collaborate on this.
>>>>> 2) I'm only aware of arrow and arrow-go currently using it.
>>>>> 
>>>>> Best,
>>>>> Rok
>>>>> 
>>>>> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
>>> wrote:
>>>>> 
>>>>>> hi all,
>>>>>> 
>>>>>> I saw that conbench.ursa.dev has been down and I had a need to set
>>> up
>>>>>> some continuous project benchmarks, and was interested in doing
>>>>>> development on Conbench (well, having my agents do development on
>>>>>> Conbench), and was interested in the following:
>>>>>> 
>>>>>> 1) is there interest in migrating the historical Arrow conbench
>> data
>>>>>> to a new server, has that been preserved somewhere? I'll probably
>>>>>> rewrite the conbench backend in Go and give it a client CLI for
>>>>>> submitting new data or querying old data.
>>>>>> 
>>>>>> 2) are there other users of conbench (conbench/conbench) that
>> anyone
>>>>>> is aware of? I'd be done doing in-situ development in that
>> repository
>>>>>> or setting up a conbench-v2 project.
>>>>>> 
>>>>>> No particular urgency but if anyone has opinions let me know!
>>>>>> 
>>>>>> thanks,
>>>>>> Wes
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Will do! Looking forward to hearing what you're looking to achieve.

Perhaps we can use the attention here (if we got any) to discuss what Arrow
needs/wants in terms of benchmarking functionality.


On Thu, Jun 4, 2026 at 9:55 PM Wes McKinney <we...@gmail.com> wrote:

> Okay sounds good! I’ll start a v2 branch and do some work on it. Can you
> contact me offline and let me know how I can get a copy of the prod
> database to help with development and test migrations?
>
> Thanks
> Wes
>
> On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:
>
> > As per my discussion with the Velox team in March, they weren't using
> > Conbench at the time and had no plans to migrate back.
> >
> > I never reported on the Conbench/Voltron migration to the mailing list,
> so
> > this is a good opportunity to do so:
> > - In November, it became clear Voltron Data would shut down its CI and
> > benchmarking infrastructure
> > - Using the donated AWS credits [1] and with the help of Mike Wendt I
> moved
> > conbench infra to the new AWS account and backfilled benchmarks to cover
> > the missing commits
> > - Raul and I moved CUDA runners to the new AWS account
> > - Moved doc-preview, Parquet benchmarking/testing datasets, the nightly
> > dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> > - Conbench gradually stabilized, but benchmarks occasionally fail due to
> > dependency issues. We don't really track these, but Nic and I typically
> fix
> > them.
> > - The database design still makes the web views slow, and we discussed
> > about dropping some history to speed them up
> > - AWS benchmark runner-minutes consume nearly the entire donated AWS
> quota.
> > Discussion on how to optimize this and what exactly we need from
> benchmarks
> > as a project occasionally comes up on Zulip, and I expect that once it
> > crystallizes we'll bring it to the ML as well.
> > - Things are now stable enough that we can discuss redesigning toward a
> > more stable and sustainable setup
> >
> > Best,
> > Rok
> >
> > [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
> >
> > On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > +1 improving conbench and the orchestration would free up aws credits
> and
> > > more importantly the time taken to fix outrages!
> > >
> > > Velox used to use conbench but through a VD provided instance. I
> haven't
> > > seen any movement to set up an independent instance.
> > >
> > >
> > > Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:
> > >
> > > > Hi Wes,
> > > >
> > > > Conbench is now at conbench.arrow-dev.org. I forked all repos to
> > > > arctosalliance github org in case VD deleted them on the way out.
> > > > Historical RDS db is preserved in the AWS account we migrated Arrow's
> > CI
> > > > infra to. It's supported by donated AWS credits.
> > > > For orchestration buildkite.com/apache-arrow is still used and I
> even
> > > got
> > > > the old M1 mac mini from Mike. It's now in my rack running benchmarks
> > > > again.
> > > >
> > > > That said current design of the orchestration layer, api server and
> db
> > is
> > > > somewhat costly and brittle. Downtime and adhoc fixes are common and
> I
> > > was
> > > > thinking about refactoring it somewhat to make it more maintainable.
> > > >
> > > > 1) If things become more maintainable and long term stable that'd be
> > > great.
> > > > I'm happy to collaborate on this.
> > > > 2) I'm only aware of arrow and arrow-go currently using it.
> > > >
> > > > Best,
> > > > Rok
> > > >
> > > > On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > hi all,
> > > > >
> > > > > I saw that conbench.ursa.dev has been down and I had a need to set
> > up
> > > > > some continuous project benchmarks, and was interested in doing
> > > > > development on Conbench (well, having my agents do development on
> > > > > Conbench), and was interested in the following:
> > > > >
> > > > > 1) is there interest in migrating the historical Arrow conbench
> data
> > > > > to a new server, has that been preserved somewhere? I'll probably
> > > > > rewrite the conbench backend in Go and give it a client CLI for
> > > > > submitting new data or querying old data.
> > > > >
> > > > > 2) are there other users of conbench (conbench/conbench) that
> anyone
> > > > > is aware of? I'd be done doing in-situ development in that
> repository
> > > > > or setting up a conbench-v2 project.
> > > > >
> > > > > No particular urgency but if anyone has opinions let me know!
> > > > >
> > > > > thanks,
> > > > > Wes
> > > > >
> > > >
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Wes McKinney <we...@gmail.com>]

Okay sounds good! I’ll start a v2 branch and do some work on it. Can you
contact me offline and let me know how I can get a copy of the prod
database to help with development and test migrations?

Thanks
Wes

On Thu, Jun 4, 2026 at 12:57 Rok Mihevc <ro...@gmail.com> wrote:

> As per my discussion with the Velox team in March, they weren't using
> Conbench at the time and had no plans to migrate back.
>
> I never reported on the Conbench/Voltron migration to the mailing list, so
> this is a good opportunity to do so:
> - In November, it became clear Voltron Data would shut down its CI and
> benchmarking infrastructure
> - Using the donated AWS credits [1] and with the help of Mike Wendt I moved
> conbench infra to the new AWS account and backfilled benchmarks to cover
> the missing commits
> - Raul and I moved CUDA runners to the new AWS account
> - Moved doc-preview, Parquet benchmarking/testing datasets, the nightly
> dashboard, etc. to S3 with help from Nic, Jacob, and Raul
> - Conbench gradually stabilized, but benchmarks occasionally fail due to
> dependency issues. We don't really track these, but Nic and I typically fix
> them.
> - The database design still makes the web views slow, and we discussed
> about dropping some history to speed them up
> - AWS benchmark runner-minutes consume nearly the entire donated AWS quota.
> Discussion on how to optimize this and what exactly we need from benchmarks
> as a project occasionally comes up on Zulip, and I expect that once it
> crystallizes we'll bring it to the ML as well.
> - Things are now stable enough that we can discuss redesigning toward a
> more stable and sustainable setup
>
> Best,
> Rok
>
> [1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv
>
> On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org>
> wrote:
>
> > +1 improving conbench and the orchestration would free up aws credits and
> > more importantly the time taken to fix outrages!
> >
> > Velox used to use conbench but through a VD provided instance. I haven't
> > seen any movement to set up an independent instance.
> >
> >
> > Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:
> >
> > > Hi Wes,
> > >
> > > Conbench is now at conbench.arrow-dev.org. I forked all repos to
> > > arctosalliance github org in case VD deleted them on the way out.
> > > Historical RDS db is preserved in the AWS account we migrated Arrow's
> CI
> > > infra to. It's supported by donated AWS credits.
> > > For orchestration buildkite.com/apache-arrow is still used and I even
> > got
> > > the old M1 mac mini from Mike. It's now in my rack running benchmarks
> > > again.
> > >
> > > That said current design of the orchestration layer, api server and db
> is
> > > somewhat costly and brittle. Downtime and adhoc fixes are common and I
> > was
> > > thinking about refactoring it somewhat to make it more maintainable.
> > >
> > > 1) If things become more maintainable and long term stable that'd be
> > great.
> > > I'm happy to collaborate on this.
> > > 2) I'm only aware of arrow and arrow-go currently using it.
> > >
> > > Best,
> > > Rok
> > >
> > > On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > hi all,
> > > >
> > > > I saw that conbench.ursa.dev has been down and I had a need to set
> up
> > > > some continuous project benchmarks, and was interested in doing
> > > > development on Conbench (well, having my agents do development on
> > > > Conbench), and was interested in the following:
> > > >
> > > > 1) is there interest in migrating the historical Arrow conbench data
> > > > to a new server, has that been preserved somewhere? I'll probably
> > > > rewrite the conbench backend in Go and give it a client CLI for
> > > > submitting new data or querying old data.
> > > >
> > > > 2) are there other users of conbench (conbench/conbench) that anyone
> > > > is aware of? I'd be done doing in-situ development in that repository
> > > > or setting up a conbench-v2 project.
> > > >
> > > > No particular urgency but if anyone has opinions let me know!
> > > >
> > > > thanks,
> > > > Wes
> > > >
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

As per my discussion with the Velox team in March, they weren't using
Conbench at the time and had no plans to migrate back.

I never reported on the Conbench/Voltron migration to the mailing list, so
this is a good opportunity to do so:
- In November, it became clear Voltron Data would shut down its CI and
benchmarking infrastructure
- Using the donated AWS credits [1] and with the help of Mike Wendt I moved
conbench infra to the new AWS account and backfilled benchmarks to cover
the missing commits
- Raul and I moved CUDA runners to the new AWS account
- Moved doc-preview, Parquet benchmarking/testing datasets, the nightly
dashboard, etc. to S3 with help from Nic, Jacob, and Raul
- Conbench gradually stabilized, but benchmarks occasionally fail due to
dependency issues. We don't really track these, but Nic and I typically fix
them.
- The database design still makes the web views slow, and we discussed
about dropping some history to speed them up
- AWS benchmark runner-minutes consume nearly the entire donated AWS quota.
Discussion on how to optimize this and what exactly we need from benchmarks
as a project occasionally comes up on Zulip, and I expect that once it
crystallizes we'll bring it to the ML as well.
- Things are now stable enough that we can discuss redesigning toward a
more stable and sustainable setup

Best,
Rok

[1] https://lists.apache.org/thread/q33oofy2v3zpg9s9l8o0w68rmjr3ocsv

On Thu, Jun 4, 2026 at 5:10 PM Jacob Wujciak <as...@apache.org> wrote:

> +1 improving conbench and the orchestration would free up aws credits and
> more importantly the time taken to fix outrages!
>
> Velox used to use conbench but through a VD provided instance. I haven't
> seen any movement to set up an independent instance.
>
>
> Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:
>
> > Hi Wes,
> >
> > Conbench is now at conbench.arrow-dev.org. I forked all repos to
> > arctosalliance github org in case VD deleted them on the way out.
> > Historical RDS db is preserved in the AWS account we migrated Arrow's CI
> > infra to. It's supported by donated AWS credits.
> > For orchestration buildkite.com/apache-arrow is still used and I even
> got
> > the old M1 mac mini from Mike. It's now in my rack running benchmarks
> > again.
> >
> > That said current design of the orchestration layer, api server and db is
> > somewhat costly and brittle. Downtime and adhoc fixes are common and I
> was
> > thinking about refactoring it somewhat to make it more maintainable.
> >
> > 1) If things become more maintainable and long term stable that'd be
> great.
> > I'm happy to collaborate on this.
> > 2) I'm only aware of arrow and arrow-go currently using it.
> >
> > Best,
> > Rok
> >
> > On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi all,
> > >
> > > I saw that conbench.ursa.dev has been down and I had a need to set up
> > > some continuous project benchmarks, and was interested in doing
> > > development on Conbench (well, having my agents do development on
> > > Conbench), and was interested in the following:
> > >
> > > 1) is there interest in migrating the historical Arrow conbench data
> > > to a new server, has that been preserved somewhere? I'll probably
> > > rewrite the conbench backend in Go and give it a client CLI for
> > > submitting new data or querying old data.
> > >
> > > 2) are there other users of conbench (conbench/conbench) that anyone
> > > is aware of? I'd be done doing in-situ development in that repository
> > > or setting up a conbench-v2 project.
> > >
> > > No particular urgency but if anyone has opinions let me know!
> > >
> > > thanks,
> > > Wes
> > >
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Jacob Wujciak <as...@apache.org>]

+1 improving conbench and the orchestration would free up aws credits and
more importantly the time taken to fix outrages!

Velox used to use conbench but through a VD provided instance. I haven't
seen any movement to set up an independent instance.


Rok Mihevc <ro...@gmail.com> schrieb am Do., 4. Juni 2026, 16:38:

> Hi Wes,
>
> Conbench is now at conbench.arrow-dev.org. I forked all repos to
> arctosalliance github org in case VD deleted them on the way out.
> Historical RDS db is preserved in the AWS account we migrated Arrow's CI
> infra to. It's supported by donated AWS credits.
> For orchestration buildkite.com/apache-arrow is still used and I even got
> the old M1 mac mini from Mike. It's now in my rack running benchmarks
> again.
>
> That said current design of the orchestration layer, api server and db is
> somewhat costly and brittle. Downtime and adhoc fixes are common and I was
> thinking about refactoring it somewhat to make it more maintainable.
>
> 1) If things become more maintainable and long term stable that'd be great.
> I'm happy to collaborate on this.
> 2) I'm only aware of arrow and arrow-go currently using it.
>
> Best,
> Rok
>
> On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi all,
> >
> > I saw that conbench.ursa.dev has been down and I had a need to set up
> > some continuous project benchmarks, and was interested in doing
> > development on Conbench (well, having my agents do development on
> > Conbench), and was interested in the following:
> >
> > 1) is there interest in migrating the historical Arrow conbench data
> > to a new server, has that been preserved somewhere? I'll probably
> > rewrite the conbench backend in Go and give it a client CLI for
> > submitting new data or querying old data.
> >
> > 2) are there other users of conbench (conbench/conbench) that anyone
> > is aware of? I'd be done doing in-situ development in that repository
> > or setting up a conbench-v2 project.
> >
> > No particular urgency but if anyone has opinions let me know!
> >
> > thanks,
> > Wes
> >
>

Re: Status of Arrow conbench data + conbench OSS project

[Rok Mihevc <ro...@gmail.com>]

Hi Wes,

Conbench is now at conbench.arrow-dev.org. I forked all repos to
arctosalliance github org in case VD deleted them on the way out.
Historical RDS db is preserved in the AWS account we migrated Arrow's CI
infra to. It's supported by donated AWS credits.
For orchestration buildkite.com/apache-arrow is still used and I even got
the old M1 mac mini from Mike. It's now in my rack running benchmarks again.

That said current design of the orchestration layer, api server and db is
somewhat costly and brittle. Downtime and adhoc fixes are common and I was
thinking about refactoring it somewhat to make it more maintainable.

1) If things become more maintainable and long term stable that'd be great.
I'm happy to collaborate on this.
2) I'm only aware of arrow and arrow-go currently using it.

Best,
Rok

On Thu, Jun 4, 2026 at 4:20 PM Wes McKinney <we...@gmail.com> wrote:

> hi all,
>
> I saw that conbench.ursa.dev has been down and I had a need to set up
> some continuous project benchmarks, and was interested in doing
> development on Conbench (well, having my agents do development on
> Conbench), and was interested in the following:
>
> 1) is there interest in migrating the historical Arrow conbench data
> to a new server, has that been preserved somewhere? I'll probably
> rewrite the conbench backend in Go and give it a client CLI for
> submitting new data or querying old data.
>
> 2) are there other users of conbench (conbench/conbench) that anyone
> is aware of? I'd be done doing in-situ development in that repository
> or setting up a conbench-v2 project.
>
> No particular urgency but if anyone has opinions let me know!
>
> thanks,
> Wes
>