You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Kazuaki Ishizaki <IS...@jp.ibm.com> on 2020/08/27 17:59:38 UTC
Ursabot Benchmark framework for other languages
I am new to the Ursabot benchmark framework like
https://github.com/apache/arrow/pull/7940#issuecomment-673183390.
How can we run benchmark programs written in other languages (e.g. Java)
beyond C++? If it is not supported yet, what do we need to support to run
benchmark programs in other languages?
Best Regards,
Kazuaki Ishizaki,
Re: Ursabot Benchmark framework for other languages
Posted by Kazuaki Ishizaki <IS...@jp.ibm.com>.
Hi François,
Thank you very much for your response with very helpful information.
I realized that this command is launched as follows at
https://ci.ursalabs.org/#/builders/73/builds/100/steps/3/logs/stdio
$ archery benchmark diff --output=diff.json
--suite-filter=parquet-encoding-benchmark
--benchmark-filter=BM_PlainDecodingBoolean --repetitions=1 WORKSPACE
origin/master
I will read these files and try to enhance "archery benchmark diff ... "
Best Regards,
Kazuaki Ishizaki
Francois Saint-Jacques <fs...@gmail.com> wrote on 2020/08/28
04:51:54:
> From: Francois Saint-Jacques <fs...@gmail.com>
> To: dev <de...@arrow.apache.org>
> Date: 2020/08/28 04:52
> Subject: [EXTERNAL] Re: Ursabot Benchmark framework for other languages
>
> Hello Kazuaki!
>
> I recommend you read and take a look at the benchmark sub-library [1]
> of archery and how it's glued [2]. You will need to implement:
>
> - A runner for the framework you intend to use [3] and [4], it also
> implies capturing the output into a class that implements the
> "Benchmark" interface.
> - Tweak the glue code such that it doesn't assume that there's only a
> CppBenchmarkRunner [5] and probably some change in the command line
> UX, e.g. `--language=<lang>` with default to cpp such that it doesn't
> break the current workflow.
>
> Once you have that and it's working locally, you can ask Krizstian
> help on how to integrate it with ursabot.
>
> François
>
> [1] INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_tree_master_dev_archery_archery_benchmark&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
>
ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=6SzHrCW51fn6u_aaWv3lVHfxjtqP69_2lzOXpfyu0Tc&e=
> [2] INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_blob_master_dev_archery_archery_cli.py-23L349-2DL577&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
>
ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=LJsnpmWSiEzpshVC-
> gKrWbgSNXlOmvi7ADX7E4qCAPw&e=
> [3] INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_blob_master_dev_archery_archery_benchmark_runner.py-23L133-2DL207&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
>
ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=jco8YNDD7bvohax6BO66r6SKbj0rUVr35ZT4Cs3ij04&e=
> [4] INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_blob_master_dev_archery_archery_benchmark_google.py-23L33-2DL175&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
>
ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=1JxymaI97EC1JGPPYUmWES0VOe5Y6y2rYwd1baXFa0g&e=
> [5] INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_blob_master_dev_archery_archery_benchmark_runner.py-23L51-2DL86&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
>
ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=Ce3HMKrtkM60N9aDCQSY-
> BAVvStfX0DKKF5c_CYGZJQ&e=
>
> On Thu, Aug 27, 2020 at 1:59 PM Kazuaki Ishizaki <IS...@jp.ibm.com>
wrote:
> >
> > I am new to the Ursabot benchmark framework like
> > INVALID URI REMOVED
>
u=https-3A__github.com_apache_arrow_pull_7940-23issuecomment-2D673183390&d=DwIFaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-
> ZCGj9Pg&m=YL3_uTOJChdZeElefIDnPIv6QJaOSuDKN7kTsLhT5PI&s=vxLT4ZdO-
> gTIqXZsLumESzBdS4XJsl9gDw3dM4cBQnc&e= .
> >
> > How can we run benchmark programs written in other languages (e.g.
Java)
> > beyond C++? If it is not supported yet, what do we need to support to
run
> > benchmark programs in other languages?
> >
> > Best Regards,
> > Kazuaki Ishizaki,
> >
>
Re: Ursabot Benchmark framework for other languages
Posted by Francois Saint-Jacques <fs...@gmail.com>.
Hello Kazuaki!
I recommend you read and take a look at the benchmark sub-library [1]
of archery and how it's glued [2]. You will need to implement:
- A runner for the framework you intend to use [3] and [4], it also
implies capturing the output into a class that implements the
"Benchmark" interface.
- Tweak the glue code such that it doesn't assume that there's only a
CppBenchmarkRunner [5] and probably some change in the command line
UX, e.g. `--language=<lang>` with default to cpp such that it doesn't
break the current workflow.
Once you have that and it's working locally, you can ask Krizstian
help on how to integrate it with ursabot.
François
[1] https://github.com/apache/arrow/tree/master/dev/archery/archery/benchmark
[2] https://github.com/apache/arrow/blob/master/dev/archery/archery/cli.py#L349-L577
[3] https://github.com/apache/arrow/blob/master/dev/archery/archery/benchmark/runner.py#L133-L207
[4] https://github.com/apache/arrow/blob/master/dev/archery/archery/benchmark/google.py#L33-L175
[5] https://github.com/apache/arrow/blob/master/dev/archery/archery/benchmark/runner.py#L51-L86
On Thu, Aug 27, 2020 at 1:59 PM Kazuaki Ishizaki <IS...@jp.ibm.com> wrote:
>
> I am new to the Ursabot benchmark framework like
> https://github.com/apache/arrow/pull/7940#issuecomment-673183390.
>
> How can we run benchmark programs written in other languages (e.g. Java)
> beyond C++? If it is not supported yet, what do we need to support to run
> benchmark programs in other languages?
>
> Best Regards,
> Kazuaki Ishizaki,
>