You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2022/08/16 13:35:50 UTC
Benchmark tests for the Beam RunInference API
Hi,
I created a doc
<https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#>[1]
which outlines the plan for the RunInference API[2] benchmark/performance
tests. I would appreciate feedback on the following,
- Models used for the benchmark tests.
- Metrics calculated as part of the benchmark tests.
If you have any inputs or any suggestions on additional metrics/models that
would be helpful for the Beam ML community as part of the benchmark tests,
please let us know.
[1]
https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#
[2]
https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269
<https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269>
Thanks,
Anand
Re: Benchmark tests for the Beam RunInference API
Posted by Andy Ye via dev <de...@beam.apache.org>.
Appreciate the super clear summary of the different benchmark experiments!
This will add lots of value to potential users, especially when we
integrate GPU benchmarks. Thanks Anand!
Best,
Andy
On Thu, Aug 18, 2022 at 10:22 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:
> I left a few comments, but overall this sounds like a good plan to me -
> thanks for the writeup!
>
> On Tue, Aug 16, 2022 at 9:36 AM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Hi,
>>
>> I created a doc
>> <https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#>[1]
>> which outlines the plan for the RunInference API[2] benchmark/performance
>> tests. I would appreciate feedback on the following,
>>
>> - Models used for the benchmark tests.
>> - Metrics calculated as part of the benchmark tests.
>>
>>
>> If you have any inputs or any suggestions on additional metrics/models
>> that would be helpful for the Beam ML community as part of the benchmark
>> tests, please let us know.
>>
>> [1]
>> https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#
>> [2]
>> https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269
>> <https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269>
>>
>>
>> Thanks,
>> Anand
>>
>
Re: Benchmark tests for the Beam RunInference API
Posted by Danny McCormick via dev <de...@beam.apache.org>.
I left a few comments, but overall this sounds like a good plan to me -
thanks for the writeup!
On Tue, Aug 16, 2022 at 9:36 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hi,
>
> I created a doc
> <https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#>[1]
> which outlines the plan for the RunInference API[2] benchmark/performance
> tests. I would appreciate feedback on the following,
>
> - Models used for the benchmark tests.
> - Metrics calculated as part of the benchmark tests.
>
>
> If you have any inputs or any suggestions on additional metrics/models
> that would be helpful for the Beam ML community as part of the benchmark
> tests, please let us know.
>
> [1]
> https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#
> [2]
> https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269
> <https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269>
>
>
> Thanks,
> Anand
>
Re: Benchmark tests for the Beam RunInference API
Posted by Danny McCormick via user <us...@beam.apache.org>.
I left a few comments, but overall this sounds like a good plan to me -
thanks for the writeup!
On Tue, Aug 16, 2022 at 9:36 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hi,
>
> I created a doc
> <https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#>[1]
> which outlines the plan for the RunInference API[2] benchmark/performance
> tests. I would appreciate feedback on the following,
>
> - Models used for the benchmark tests.
> - Metrics calculated as part of the benchmark tests.
>
>
> If you have any inputs or any suggestions on additional metrics/models
> that would be helpful for the Beam ML community as part of the benchmark
> tests, please let us know.
>
> [1]
> https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#
> [2]
> https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269
> <https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269>
>
>
> Thanks,
> Anand
>