You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by QP Hou <ho...@apache.org> on 2021/09/11 03:15:27 UTC
[DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Hi,
I think conbench.ursa.dev works really well for the main arrow repo,
especially the ability to request on demand benchmarks during PR
reviews by mentioning usrabot in a comment.
I am wondering if it is something that arrow-rs and arrow-datafusion
could leverage as well to help speed up PR reviews and guard against
performance degradation? If so, what's the process for setting it up?
Thanks,
QP
RE: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by Matthew Topol <mt...@factset.com>.
This info is really useful for other languages that want to publish to conbench also, I'd love to find some time to try to also publish the Go benchmarks there too, luckily the go impl is already in the main arrow repo, so other than figuring out how to do the publishing, I'd just need Wes' blessing.
If I manage to find some time, I'll probably look into this more seriously for the Go implementation. Would it make sense to have something actually published somewhere that is more updated since the blog post is outdated now?
-----Original Message-----
From: Diana Clarke <di...@gmail.com>
Sent: Saturday, September 11, 2021 11:37 AM
To: dev@arrow.apache.org
Subject: Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
I'd love to see arrow-rs and arrow-datafusion also publish results to Arrow's Conbench server.
If you point me to the existing benchmarks for each project and instructions on how to execute them, I can let you know the easiest integration path.
If the arrow-rs benchmarks are executable from command line and return parsable results (like json), it should be pretty easy to publish the results.
And if any of the existing benchmarks are written in Python (perhaps arrow-datafusion?), then there may be a different (but also easy) integration path.
There are a few caveats though:
- The benchmarks are running on dedicated machines at Wes' house, so we would need his blessing (or additional dedicated machines).
- The arrow-rs and arrow-datafusion GitHub repositories must use squash merges (or Conbench would have to be extended to understand the
2 other GitHub merge methods).
- I'm not sure what the security implications are with respect to adding our ursabot integration and buildkite hooks to other repositories.
In the meantime, here's a blog post that initially announced the Arrow/Conbench integration and contains a brief functionality overview.
https://urldefense.com/v3/__https://ursalabs.org/blog/announcing-conbench/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SishwuMxk$
That blog post is a bit outdated now – we've since started publishing the Arrow Java and JavaScript benchmarks to https://urldefense.com/v3/__https://conbench.ursa.dev/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiMWTIttY$
as well.
And Conbench now collects and plots benchmark distribution history, and computes z-scores (rather than just a basic percent changed).
Example distribution history (note the regression):
https://urldefense.com/v3/__https://conbench.ursa.dev/compare/benchmarks/7efda5a661a348ebb66e021dc9aa7bba...6a4122865bba4b6b9045390d4d1eb7cc/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiHXS6EbM$
Example simple benchmark result plotting:
https://urldefense.com/v3/__https://conbench.ursa.dev/batches/cbe57f13e51f46019f207853421db764/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SinbmHAmQ$
Example run of the Arrow C++ and Java benchmarks:
https://urldefense.com/v3/__https://conbench.ursa.dev/runs/82308256f17a4d059db902b07d4d4530/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiXPdw6A4$
Example run of the Arrow R, Python, and JavaScript benchmarks:
https://urldefense.com/v3/__https://conbench.ursa.dev/runs/b56fdd1375ed4b0cbf928dd8dafe8322/__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiWBdn2lw$
Finally, here's a link to the Conbench repository (as well as the repositories with the Arrow benchmarks):
- Conbench:
https://urldefense.com/v3/__https://github.com/ursacomputing/conbench__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiTUlgkB0$
- Arrow Python benchmarks + external benchmark wrappers/runners:
https://urldefense.com/v3/__https://github.com/ursacomputing/benchmarks__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SifALMAmM$
- Arrow R benchmarks:
https://urldefense.com/v3/__https://github.com/ursacomputing/arrowbench__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8SiRrJfhbg$
- Arrow Java/JavaScript/C++ benchmarks:
https://urldefense.com/v3/__https://github.com/apache/arrow__;!!PBKjc0U4!f8Ne8f-5H7C3l7AYnYDUimJsSvOmLGizeQVAftXT7AMHkfyf7QOZl8Sizr5ywCE$
Looking forward to collaborating on this!
--diana
On Sat, Sep 11, 2021 at 5:01 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> I wasn't even aware of conbench.ursa.dev -- I agree it looks like a
> really neat tool and could be very valuable for the rust language
> implementation
>
> On Fri, Sep 10, 2021 at 11:16 PM QP Hou <ho...@apache.org> wrote:
>
> > Hi,
> >
> > I think conbench.ursa.dev works really well for the main arrow repo,
> > especially the ability to request on demand benchmarks during PR
> > reviews by mentioning usrabot in a comment.
> >
> > I am wondering if it is something that arrow-rs and arrow-datafusion
> > could leverage as well to help speed up PR reviews and guard against
> > performance degradation? If so, what's the process for setting it up?
> >
> > Thanks,
> > QP
> >
Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by QP Hou <ho...@gmail.com>.
Thank you Diana for the quick turnaround! The trail run looks great.
You are right that `sqrt_20_12, sqrt_20_9, sqrt_22_12, and sqrt_22_14`
are just the same type of test with different parameters, so it makes
sense to batch them. We can name these benchmarks however we want to
make it easier for conbench to parse the metadata.
I am able to reproduce the error you got with arrow-rs. I have filed
https://github.com/apache/arrow-rs/issues/770 to track the build
issue. In the meantime, you can workaround it by running the `cargo
bench` command within the arrow folder to avoid building benchmarks
from the parquet trait.
> - We probably want to add some additional context, like the
arrow-rs/arrow-datafusion version, rust version, any compiler flags,
etc.
I agree, the rust version can be obtained through `rustc --version`.
We are not setting any special compiler flag at the moment, but
definitely something useful to include if we are planning to add any
in the future.
Please don't hesitate to reach out if there is anything we could help
to unblock you. For synchronous communication, most of the Rust
developers are active in the Apache slack workspace's #arrow-rust
channel.
On Sun, Sep 12, 2021 at 12:11 PM Diana Clarke
<di...@gmail.com> wrote:
>
> Thanks for those great instructions, QP!
>
> I've spiked adding the arrow-rs and arrow-datafusion benchmark results
> to the Arrow Conbench server.
>
> https://github.com/ursacomputing/benchmarks/pull/79/files
>
> And I've published one example run for arrow-datafusion:
>
> - Example run: https://conbench.ursa.dev/runs/62098d2f86314339a696f8c48c4ce2e7/
> - Example benchmark from run:
> https://conbench.ursa.dev/benchmarks/aa841cdf78ef4e38832e075b8485ec59/
>
> Some observations along the way:
>
> - Criterion results are in nanoseconds, but the smallest unit
> Conbench currently speaks is seconds (because Conbench was initially
> for macro not micro benchmarking). I suspect most places in Conbench
> would work just fine if nanoseconds were passed in, but I need to
> audit the code for any places that assume seconds if it isn't a
> throughput benchmark.
>
> - If the Criterion benchmarks were named better, I could tag them
> better in Conbench. For example, I suspect sqrt_20_12, sqrt_20_9,
> sqrt_22_12, and sqrt_22_14 are parameterized variations of the same
> benchmark, and if they were named something like "sqrt, foo=20,
> bar=12", I could batch them together & tag their parameters so that
> Conbench would automatically graph them in relation to each other. I
> was sort of able to do this with the following benchmarks (because
> there was a machine readable pattern). Anyhoo, that's easy enough to
> do down the road as a last integration step, and it does appear from
> the Criterion docs that they have their own recommendations for how to
> do this.
>
> - window partition by, u64_narrow, aggregate functions
> - window partition by, u64_narrow, built-in functions
> - window partition by, u64_wide, aggregate functions
> - window partition by, u64_wide, built-in functions
>
> - While Criterion benchmarks can also measure throughput in some
> cases, all the arrow-datafusion benchmarks were in elapsed time (not
> sure about the arrow-rs benchmarks), so I didn't bother writing code
> to support potential throughput results from
> arrow-datafusion/arrow-rs, but we may need to revisit that.
>
> - We probably want to add some additional context, like the
> arrow-rs/arrow-datafusion version, rust version, any compiler flags,
> etc.
>
> PS. I wasn't able to get the arrow-rs benchmarks to run, but it sounds
> like they are very similar to arrow-datafusion benchmarks. I don't
> know Rust, so I may reach out on zulip (or wherever Arrow Rust folks
> talk) for help building arrow-rs.
>
> $ cargo bench
> unresolved imports `parquet::util::DataPageBuilder`,
> `parquet::util::DataPageBuilderImpl`,
> `parquet::util::InMemoryPageIterator`
>
> There are still the orchestration steps that need to be worked on, but
> all-in-all this seems very doable. I just need to negotiate some time
> with my day-job.
>
> Cheers,
>
> --diana
>
> On Sat, Sep 11, 2021 at 4:35 PM QP Hou <ho...@gmail.com> wrote:
> >
> > Thanks Diana a lot for offering to help. Please see my replies inline below.
> >
> > On Sat, Sep 11, 2021 at 8:37 AM Diana Clarke
> > <di...@gmail.com> wrote:
> > > If you point me to the existing benchmarks for each project and
> > > instructions on how to execute them, I can let you know the easiest
> > > integration path.
> > >
> >
> > For arrow-datafusion, you just need to install the rust toolchain
> > using `rustup` [1], then run the `cargo bench` command within the
> > project root. Benchmark results will be saved under the
> > `/target/criterion/BENCH_NAME/new` folder as raw.csv file. You can
> > read more about the convention at
> > https://bheisler.github.io/criterion.rs/book/user_guide/csv_output.html.
> >
> > For arrow-rs, it's the exact same setup.
> >
> > We have some extra TPCH integration benchmarks in datafusion, but I
> > think we can work on integrating them later. Getting the basic
> > criterion benchmarks into conbench would already be a huge win for us.
> >
> > [1]: https://rustup.rs
> >
> > > If the arrow-rs benchmarks are executable from command line and return
> > > parsable results (like json), it should be pretty easy to publish the
> > > results.
> > >
> >
> > By default, results are saved as csv files, but you can pass in a
> > `--message-format=json` argument to save the results as JSON files
> > instead.
> >
> > > - The arrow-rs and arrow-datafusion GitHub repositories must use
> > > squash merges (or Conbench would have to be extended to understand the
> > > 2 other GitHub merge methods).
> >
> > Yes, we are using squash merge for both repos.
> >
> > > - I'm not sure what the security implications are with respect to
> > > adding our ursabot integration and buildkite hooks to other
> > > repositories.
> > >
> >
> > Are you concerned about security from ursabot and buildkit's point of
> > views? If so, who should we reach out to discuss this matter?
> >
> > Thanks,
> > QP
Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by Diana Clarke <di...@gmail.com>.
Thanks for those great instructions, QP!
I've spiked adding the arrow-rs and arrow-datafusion benchmark results
to the Arrow Conbench server.
https://github.com/ursacomputing/benchmarks/pull/79/files
And I've published one example run for arrow-datafusion:
- Example run: https://conbench.ursa.dev/runs/62098d2f86314339a696f8c48c4ce2e7/
- Example benchmark from run:
https://conbench.ursa.dev/benchmarks/aa841cdf78ef4e38832e075b8485ec59/
Some observations along the way:
- Criterion results are in nanoseconds, but the smallest unit
Conbench currently speaks is seconds (because Conbench was initially
for macro not micro benchmarking). I suspect most places in Conbench
would work just fine if nanoseconds were passed in, but I need to
audit the code for any places that assume seconds if it isn't a
throughput benchmark.
- If the Criterion benchmarks were named better, I could tag them
better in Conbench. For example, I suspect sqrt_20_12, sqrt_20_9,
sqrt_22_12, and sqrt_22_14 are parameterized variations of the same
benchmark, and if they were named something like "sqrt, foo=20,
bar=12", I could batch them together & tag their parameters so that
Conbench would automatically graph them in relation to each other. I
was sort of able to do this with the following benchmarks (because
there was a machine readable pattern). Anyhoo, that's easy enough to
do down the road as a last integration step, and it does appear from
the Criterion docs that they have their own recommendations for how to
do this.
- window partition by, u64_narrow, aggregate functions
- window partition by, u64_narrow, built-in functions
- window partition by, u64_wide, aggregate functions
- window partition by, u64_wide, built-in functions
- While Criterion benchmarks can also measure throughput in some
cases, all the arrow-datafusion benchmarks were in elapsed time (not
sure about the arrow-rs benchmarks), so I didn't bother writing code
to support potential throughput results from
arrow-datafusion/arrow-rs, but we may need to revisit that.
- We probably want to add some additional context, like the
arrow-rs/arrow-datafusion version, rust version, any compiler flags,
etc.
PS. I wasn't able to get the arrow-rs benchmarks to run, but it sounds
like they are very similar to arrow-datafusion benchmarks. I don't
know Rust, so I may reach out on zulip (or wherever Arrow Rust folks
talk) for help building arrow-rs.
$ cargo bench
unresolved imports `parquet::util::DataPageBuilder`,
`parquet::util::DataPageBuilderImpl`,
`parquet::util::InMemoryPageIterator`
There are still the orchestration steps that need to be worked on, but
all-in-all this seems very doable. I just need to negotiate some time
with my day-job.
Cheers,
--diana
On Sat, Sep 11, 2021 at 4:35 PM QP Hou <ho...@gmail.com> wrote:
>
> Thanks Diana a lot for offering to help. Please see my replies inline below.
>
> On Sat, Sep 11, 2021 at 8:37 AM Diana Clarke
> <di...@gmail.com> wrote:
> > If you point me to the existing benchmarks for each project and
> > instructions on how to execute them, I can let you know the easiest
> > integration path.
> >
>
> For arrow-datafusion, you just need to install the rust toolchain
> using `rustup` [1], then run the `cargo bench` command within the
> project root. Benchmark results will be saved under the
> `/target/criterion/BENCH_NAME/new` folder as raw.csv file. You can
> read more about the convention at
> https://bheisler.github.io/criterion.rs/book/user_guide/csv_output.html.
>
> For arrow-rs, it's the exact same setup.
>
> We have some extra TPCH integration benchmarks in datafusion, but I
> think we can work on integrating them later. Getting the basic
> criterion benchmarks into conbench would already be a huge win for us.
>
> [1]: https://rustup.rs
>
> > If the arrow-rs benchmarks are executable from command line and return
> > parsable results (like json), it should be pretty easy to publish the
> > results.
> >
>
> By default, results are saved as csv files, but you can pass in a
> `--message-format=json` argument to save the results as JSON files
> instead.
>
> > - The arrow-rs and arrow-datafusion GitHub repositories must use
> > squash merges (or Conbench would have to be extended to understand the
> > 2 other GitHub merge methods).
>
> Yes, we are using squash merge for both repos.
>
> > - I'm not sure what the security implications are with respect to
> > adding our ursabot integration and buildkite hooks to other
> > repositories.
> >
>
> Are you concerned about security from ursabot and buildkit's point of
> views? If so, who should we reach out to discuss this matter?
>
> Thanks,
> QP
Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by QP Hou <ho...@gmail.com>.
Thanks Diana a lot for offering to help. Please see my replies inline below.
On Sat, Sep 11, 2021 at 8:37 AM Diana Clarke
<di...@gmail.com> wrote:
> If you point me to the existing benchmarks for each project and
> instructions on how to execute them, I can let you know the easiest
> integration path.
>
For arrow-datafusion, you just need to install the rust toolchain
using `rustup` [1], then run the `cargo bench` command within the
project root. Benchmark results will be saved under the
`/target/criterion/BENCH_NAME/new` folder as raw.csv file. You can
read more about the convention at
https://bheisler.github.io/criterion.rs/book/user_guide/csv_output.html.
For arrow-rs, it's the exact same setup.
We have some extra TPCH integration benchmarks in datafusion, but I
think we can work on integrating them later. Getting the basic
criterion benchmarks into conbench would already be a huge win for us.
[1]: https://rustup.rs
> If the arrow-rs benchmarks are executable from command line and return
> parsable results (like json), it should be pretty easy to publish the
> results.
>
By default, results are saved as csv files, but you can pass in a
`--message-format=json` argument to save the results as JSON files
instead.
> - The arrow-rs and arrow-datafusion GitHub repositories must use
> squash merges (or Conbench would have to be extended to understand the
> 2 other GitHub merge methods).
Yes, we are using squash merge for both repos.
> - I'm not sure what the security implications are with respect to
> adding our ursabot integration and buildkite hooks to other
> repositories.
>
Are you concerned about security from ursabot and buildkit's point of
views? If so, who should we reach out to discuss this matter?
Thanks,
QP
Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by Diana Clarke <di...@gmail.com>.
I'd love to see arrow-rs and arrow-datafusion also publish results to
Arrow's Conbench server.
If you point me to the existing benchmarks for each project and
instructions on how to execute them, I can let you know the easiest
integration path.
If the arrow-rs benchmarks are executable from command line and return
parsable results (like json), it should be pretty easy to publish the
results.
And if any of the existing benchmarks are written in Python (perhaps
arrow-datafusion?), then there may be a different (but also easy)
integration path.
There are a few caveats though:
- The benchmarks are running on dedicated machines at Wes' house, so
we would need his blessing (or additional dedicated machines).
- The arrow-rs and arrow-datafusion GitHub repositories must use
squash merges (or Conbench would have to be extended to understand the
2 other GitHub merge methods).
- I'm not sure what the security implications are with respect to
adding our ursabot integration and buildkite hooks to other
repositories.
In the meantime, here's a blog post that initially announced the
Arrow/Conbench integration and contains a brief functionality
overview.
https://ursalabs.org/blog/announcing-conbench/
That blog post is a bit outdated now – we've since started publishing
the Arrow Java and JavaScript benchmarks to https://conbench.ursa.dev/
as well.
And Conbench now collects and plots benchmark distribution history,
and computes z-scores (rather than just a basic percent changed).
Example distribution history (note the regression):
https://conbench.ursa.dev/compare/benchmarks/7efda5a661a348ebb66e021dc9aa7bba...6a4122865bba4b6b9045390d4d1eb7cc/
Example simple benchmark result plotting:
https://conbench.ursa.dev/batches/cbe57f13e51f46019f207853421db764/
Example run of the Arrow C++ and Java benchmarks:
https://conbench.ursa.dev/runs/82308256f17a4d059db902b07d4d4530/
Example run of the Arrow R, Python, and JavaScript benchmarks:
https://conbench.ursa.dev/runs/b56fdd1375ed4b0cbf928dd8dafe8322/
Finally, here's a link to the Conbench repository (as well as the
repositories with the Arrow benchmarks):
- Conbench:
https://github.com/ursacomputing/conbench
- Arrow Python benchmarks + external benchmark wrappers/runners:
https://github.com/ursacomputing/benchmarks
- Arrow R benchmarks:
https://github.com/ursacomputing/arrowbench
- Arrow Java/JavaScript/C++ benchmarks:
https://github.com/apache/arrow
Looking forward to collaborating on this!
--diana
On Sat, Sep 11, 2021 at 5:01 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> I wasn't even aware of conbench.ursa.dev -- I agree it looks like a really
> neat tool and could be very valuable for the rust language implementation
>
> On Fri, Sep 10, 2021 at 11:16 PM QP Hou <ho...@apache.org> wrote:
>
> > Hi,
> >
> > I think conbench.ursa.dev works really well for the main arrow repo,
> > especially the ability to request on demand benchmarks during PR
> > reviews by mentioning usrabot in a comment.
> >
> > I am wondering if it is something that arrow-rs and arrow-datafusion
> > could leverage as well to help speed up PR reviews and guard against
> > performance degradation? If so, what's the process for setting it up?
> >
> > Thanks,
> > QP
> >
Re: [DISCUSS] Use conbench.ursa.dev for arrow-rs and arrow-datafusion
Posted by Andrew Lamb <al...@influxdata.com>.
I wasn't even aware of conbench.ursa.dev -- I agree it looks like a really
neat tool and could be very valuable for the rust language implementation
On Fri, Sep 10, 2021 at 11:16 PM QP Hou <ho...@apache.org> wrote:
> Hi,
>
> I think conbench.ursa.dev works really well for the main arrow repo,
> especially the ability to request on demand benchmarks during PR
> reviews by mentioning usrabot in a comment.
>
> I am wondering if it is something that arrow-rs and arrow-datafusion
> could leverage as well to help speed up PR reviews and guard against
> performance degradation? If so, what's the process for setting it up?
>
> Thanks,
> QP
>