You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Dariusz Aniszewski <da...@polidea.com> on 2018/04/13 16:02:33 UTC
Performance tests status and anomaly detection proposal
*Hello Beam devs!As you might already noticed, together with Łukasz Gajowy,
Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here) we’re
working on adding some performance tests to the project. We were following
directions from the Testing I/O Transforms in Apache Beam
<https://beam.apache.org/documentation/io/testing/> site (which we plan to
update in near future).We started from testing various FileBasedIOs as part
of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
for TFRecordIO, see BEAM-3945
<https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
was there, but was disabled; we fixed it and enabled)- HCatalogIO
(currently in PR <https://github.com/apache/beam/pull/5097>)While currently
all the tests are maven-based, we responded to ongoing Gradle migration and
created PR <https://github.com/apache/beam/pull/5003> that allows running
them via Gradle.All of those tests are executed on daily basis on Apache
Jenkins <https://builds.apache.org/> and their results are published to
individual BigQuery tables. There is also a dashboard on which tests
results may be viewed and
compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
<https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
we have some amount of tests already, we’re currently working on a tool
that will analyze the results and search for anomalies, so devs are
notified if degraded performance is observed. You can find proposal
document
here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
<https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
welcome you to share your thoughts on performance tests in general as well
as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
Re: Performance tests status and anomaly detection proposal
Posted by Tim Robertson <ti...@gmail.com>.
Very nice Dariusz
+1 to putting the daily report on Slack to help prompt early comments and
investigation on change.
A couple very minor comments:
a) Would it make sense to track number of tests run as well as duration?
b) Labels on the Y-axis / a title with units would be a good addition to
the charts (sorry, bugbear of mine)
I've recently added retry-on-failure code to SolrIO and the tests required
doing things like verifying the timeout is observed. This increased the
test runtime, and your alerting to all devs would help communicate that and
hopefully prompt folk to raise any concerns early.
Thanks for sharing,
Tim
On Sun, Apr 15, 2018 at 1:20 AM, Kenneth Knowles <kl...@google.com> wrote:
> This is very cool. So is it easy for someone to integrate the proposal to
> regularly run Nexmark benchmarks and get those on the dashboard? (or a
> separate one to keep IOs in their own page)
>
> Kenn
>
> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
> dariusz.aniszewski@polidea.com> wrote:
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Hello Beam devs!As you might already noticed, together with Łukasz
>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>> we’re working on adding some performance tests to the project. We were
>> following directions from the Testing I/O Transforms in Apache Beam
>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>> update in near future).We started from testing various FileBasedIOs as part
>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>> for TFRecordIO, see BEAM-3945
>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>> all the tests are maven-based, we responded to ongoing Gradle migration and
>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>> them via Gradle.All of those tests are executed on daily basis on Apache
>> Jenkins <https://builds.apache.org/> and their results are published to
>> individual BigQuery tables. There is also a dashboard on which tests
>> results may be viewed and
>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>> we have some amount of tests already, we’re currently working on a tool
>> that will analyze the results and search for anomalies, so devs are
>> notified if degraded performance is observed. You can find proposal
>> document
>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>> welcome you to share your thoughts on performance tests in general as well
>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>
>
Re: Performance tests status and anomaly detection proposal
Posted by Kamil Szewczyk <sz...@gmail.com>.
Thanks for reviewing it. When operating on average there should also be
presented standard deviation. Higher deviation means that we have more
distributed and extreme results results and lower deviation value means
that results are closer the average. It will indicate that tests are going
more stable, by having values closer to trend. That also indicate that
occurrence of any anomalies is rather not a false positive. However would
be really nice to run some queries over real data exported from BigQuery,
and see which statistic parameters could also be useful.
2018-04-16 19:38 GMT+02:00 Jason Kuster <ja...@google.com>:
> Great suggestions -- added some comments. Do you have plans to add more
> sophisticated analysis past just analyzing runtime relative to the 6d
> average?
>
> On Mon, Apr 16, 2018 at 10:21 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>> That is correct - for now, we're running tests on Dataflow only. There
>> were plans to run them on Spark and Flink (and possibly more runners) but
>> we had some difficulties on the way. We decided to focus on Dataflow, at
>> least for now. Currently the tests are quite flaky, so the priority is to
>> make them more stable. Meanwhile we're providing the all necessary
>> "infrastructure" (hence the anomaly detection proposal).
>>
>> If anyone is willing to contribute in this area, those seem to be the
>> biggest blockers for Spark and Flink:
>> https://issues.apache.org/jira/browse/BEAM-3370
>> https://issues.apache.org/jira/browse/BEAM-3371
>>
>> Best regards,
>> Łukasz
>>
>> 2018-04-16 18:28 GMT+02:00 Pablo Estrada <pa...@google.com>:
>>
>>> This is very cool!
>>> Are these dashboards for tests running on Dataflow only? Are there plans
>>> for other runners? : )
>>> -P.
>>>
>>> On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>> Thanks Dariusz. This sounds great. Added some comments.
>>>> Also, +Jeff Gardner <ga...@google.com> who has experience on
>>>> performance regression analysis of integration tests.
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>>
>>>> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
>>>> wrote:
>>>>
>>>>> @Etienne +1 to doing that! :) if we have both results (Nexmark and
>>>>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies
>>>>> captured by Nexmark (if there's need for doing that).
>>>>>
>>>>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>>>>>
>>>>>> Very nice to see the dashboards !
>>>>>>
>>>>>> Regarding Kenn's comment: Nexmark supports outputing the results to
>>>>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>>>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>>>>> dashboard tables ?
>>>>>> WDYT?
>>>>>>
>>>>>> Etienne
>>>>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>>>>
>>>>>> This is very cool. So is it easy for someone to integrate the
>>>>>> proposal to regularly run Nexmark benchmarks and get those on the
>>>>>> dashboard? (or a separate one to keep IOs in their own page)
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>>>>> dariusz.aniszewski@polidea.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>>>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>>>>> we’re working on adding some performance tests to the project. We were
>>>>>> following directions from the Testing I/O Transforms in Apache Beam
>>>>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>>>>> update in near future).We started from testing various FileBasedIOs as part
>>>>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>>>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>>>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>>>>>> for TFRecordIO, see BEAM-3945
>>>>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>>>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>>>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>>>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>>>>>> all the tests are maven-based, we responded to ongoing Gradle migration and
>>>>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>>>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>>>>> Jenkins <https://builds.apache.org/> and their results are published to
>>>>>> individual BigQuery tables. There is also a dashboard on which tests
>>>>>> results may be viewed and
>>>>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>>>>> we have some amount of tests already, we’re currently working on a tool
>>>>>> that will analyze the results and search for anomalies, so devs are
>>>>>> notified if degraded performance is observed. You can find proposal
>>>>>> document
>>>>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>>>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>>>>> welcome you to share your thoughts on performance tests in general as well
>>>>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>>>>
>>>>>>
>>>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
>>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
>
Re: Performance tests status and anomaly detection proposal
Posted by Jason Kuster <ja...@google.com>.
Great suggestions -- added some comments. Do you have plans to add more
sophisticated analysis past just analyzing runtime relative to the 6d
average?
On Mon, Apr 16, 2018 at 10:21 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
> That is correct - for now, we're running tests on Dataflow only. There
> were plans to run them on Spark and Flink (and possibly more runners) but
> we had some difficulties on the way. We decided to focus on Dataflow, at
> least for now. Currently the tests are quite flaky, so the priority is to
> make them more stable. Meanwhile we're providing the all necessary
> "infrastructure" (hence the anomaly detection proposal).
>
> If anyone is willing to contribute in this area, those seem to be the
> biggest blockers for Spark and Flink:
> https://issues.apache.org/jira/browse/BEAM-3370
> https://issues.apache.org/jira/browse/BEAM-3371
>
> Best regards,
> Łukasz
>
> 2018-04-16 18:28 GMT+02:00 Pablo Estrada <pa...@google.com>:
>
>> This is very cool!
>> Are these dashboards for tests running on Dataflow only? Are there plans
>> for other runners? : )
>> -P.
>>
>> On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Thanks Dariusz. This sounds great. Added some comments.
>>> Also, +Jeff Gardner <ga...@google.com> who has experience on
>>> performance regression analysis of integration tests.
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
>>> wrote:
>>>
>>>> @Etienne +1 to doing that! :) if we have both results (Nexmark and
>>>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies
>>>> captured by Nexmark (if there's need for doing that).
>>>>
>>>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>>>>
>>>>> Very nice to see the dashboards !
>>>>>
>>>>> Regarding Kenn's comment: Nexmark supports outputing the results to
>>>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>>>> dashboard tables ?
>>>>> WDYT?
>>>>>
>>>>> Etienne
>>>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>>>
>>>>> This is very cool. So is it easy for someone to integrate the proposal
>>>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a
>>>>> separate one to keep IOs in their own page)
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>>>> dariusz.aniszewski@polidea.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>>>> we’re working on adding some performance tests to the project. We were
>>>>> following directions from the Testing I/O Transforms in Apache Beam
>>>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>>>> update in near future).We started from testing various FileBasedIOs as part
>>>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>>>>> for TFRecordIO, see BEAM-3945
>>>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>>>>> all the tests are maven-based, we responded to ongoing Gradle migration and
>>>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>>>> Jenkins <https://builds.apache.org/> and their results are published to
>>>>> individual BigQuery tables. There is also a dashboard on which tests
>>>>> results may be viewed and
>>>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>>>> we have some amount of tests already, we’re currently working on a tool
>>>>> that will analyze the results and search for anomalies, so devs are
>>>>> notified if degraded performance is observed. You can find proposal
>>>>> document
>>>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>>>> welcome you to share your thoughts on performance tests in general as well
>>>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>>>
>>>>>
>>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
>
--
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow
See something? Say something. go/jasonkuster-feedback
Re: Performance tests status and anomaly detection proposal
Posted by Łukasz Gajowy <lu...@gmail.com>.
That is correct - for now, we're running tests on Dataflow only. There were
plans to run them on Spark and Flink (and possibly more runners) but we had
some difficulties on the way. We decided to focus on Dataflow, at least for
now. Currently the tests are quite flaky, so the priority is to make them
more stable. Meanwhile we're providing the all necessary "infrastructure"
(hence the anomaly detection proposal).
If anyone is willing to contribute in this area, those seem to be the
biggest blockers for Spark and Flink:
https://issues.apache.org/jira/browse/BEAM-3370
https://issues.apache.org/jira/browse/BEAM-3371
Best regards,
Łukasz
2018-04-16 18:28 GMT+02:00 Pablo Estrada <pa...@google.com>:
> This is very cool!
> Are these dashboards for tests running on Dataflow only? Are there plans
> for other runners? : )
> -P.
>
> On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Thanks Dariusz. This sounds great. Added some comments.
>> Also, +Jeff Gardner <ga...@google.com> who has experience on
>> performance regression analysis of integration tests.
>>
>> Thanks,
>> Cham
>>
>>
>> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> @Etienne +1 to doing that! :) if we have both results (Nexmark and
>>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies
>>> captured by Nexmark (if there's need for doing that).
>>>
>>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>>>
>>>> Very nice to see the dashboards !
>>>>
>>>> Regarding Kenn's comment: Nexmark supports outputing the results to
>>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>>> dashboard tables ?
>>>> WDYT?
>>>>
>>>> Etienne
>>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>>
>>>> This is very cool. So is it easy for someone to integrate the proposal
>>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a
>>>> separate one to keep IOs in their own page)
>>>>
>>>> Kenn
>>>>
>>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>>> dariusz.aniszewski@polidea.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>>> we’re working on adding some performance tests to the project. We were
>>>> following directions from the Testing I/O Transforms in Apache Beam
>>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>>> update in near future).We started from testing various FileBasedIOs as part
>>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>>>> for TFRecordIO, see BEAM-3945
>>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>>>> all the tests are maven-based, we responded to ongoing Gradle migration and
>>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>>> Jenkins <https://builds.apache.org/> and their results are published to
>>>> individual BigQuery tables. There is also a dashboard on which tests
>>>> results may be viewed and
>>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>>> we have some amount of tests already, we’re currently working on a tool
>>>> that will analyze the results and search for anomalies, so devs are
>>>> notified if degraded performance is observed. You can find proposal
>>>> document
>>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>>> welcome you to share your thoughts on performance tests in general as well
>>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>>
>>>>
>>> --
> Got feedback? go/pabloem-feedback
>
Re: Performance tests status and anomaly detection proposal
Posted by Kenneth Knowles <kl...@google.com>.
Cool. I filed https://issues.apache.org/jira/browse/BEAM-4089 to track the
idea of reporting Nexmark runs to the dashboard.
Kenn
On Mon, Apr 16, 2018 at 9:29 AM Pablo Estrada <pa...@google.com> wrote:
> This is very cool!
> Are these dashboards for tests running on Dataflow only? Are there plans
> for other runners? : )
> -P.
>
> On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Thanks Dariusz. This sounds great. Added some comments.
>> Also, +Jeff Gardner <ga...@google.com> who has experience on
>> performance regression analysis of integration tests.
>>
>> Thanks,
>> Cham
>>
>>
>> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> @Etienne +1 to doing that! :) if we have both results (Nexmark and
>>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies
>>> captured by Nexmark (if there's need for doing that).
>>>
>>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>>>
>>>> Very nice to see the dashboards !
>>>>
>>>> Regarding Kenn's comment: Nexmark supports outputing the results to
>>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>>> dashboard tables ?
>>>> WDYT?
>>>>
>>>> Etienne
>>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>>
>>>> This is very cool. So is it easy for someone to integrate the proposal
>>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a
>>>> separate one to keep IOs in their own page)
>>>>
>>>> Kenn
>>>>
>>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>>> dariusz.aniszewski@polidea.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>>> we’re working on adding some performance tests to the project. We were
>>>> following directions from the Testing I/O Transforms in Apache Beam
>>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>>> update in near future).We started from testing various FileBasedIOs as part
>>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>>>> for TFRecordIO, see BEAM-3945
>>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>>>> all the tests are maven-based, we responded to ongoing Gradle migration and
>>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>>> Jenkins <https://builds.apache.org/> and their results are published to
>>>> individual BigQuery tables. There is also a dashboard on which tests
>>>> results may be viewed and
>>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>>> we have some amount of tests already, we’re currently working on a tool
>>>> that will analyze the results and search for anomalies, so devs are
>>>> notified if degraded performance is observed. You can find proposal
>>>> document
>>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>>> welcome you to share your thoughts on performance tests in general as well
>>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>>
>>>>
>>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: Performance tests status and anomaly detection proposal
Posted by Pablo Estrada <pa...@google.com>.
This is very cool!
Are these dashboards for tests running on Dataflow only? Are there plans
for other runners? : )
-P.
On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <ch...@google.com>
wrote:
> Thanks Dariusz. This sounds great. Added some comments.
> Also, +Jeff Gardner <ga...@google.com> who has experience on
> performance regression analysis of integration tests.
>
> Thanks,
> Cham
>
>
> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>> @Etienne +1 to doing that! :) if we have both results (Nexmark and IOITs)
>> in BQ we could use the same (similar?) tools to detect anomalies captured
>> by Nexmark (if there's need for doing that).
>>
>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>>
>>> Very nice to see the dashboards !
>>>
>>> Regarding Kenn's comment: Nexmark supports outputing the results to
>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>> dashboard tables ?
>>> WDYT?
>>>
>>> Etienne
>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>
>>> This is very cool. So is it easy for someone to integrate the proposal
>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a
>>> separate one to keep IOs in their own page)
>>>
>>> Kenn
>>>
>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>> dariusz.aniszewski@polidea.com> wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>> we’re working on adding some performance tests to the project. We were
>>> following directions from the Testing I/O Transforms in Apache Beam
>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>> update in near future).We started from testing various FileBasedIOs as part
>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>>> for TFRecordIO, see BEAM-3945
>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>>> all the tests are maven-based, we responded to ongoing Gradle migration and
>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>> Jenkins <https://builds.apache.org/> and their results are published to
>>> individual BigQuery tables. There is also a dashboard on which tests
>>> results may be viewed and
>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>> we have some amount of tests already, we’re currently working on a tool
>>> that will analyze the results and search for anomalies, so devs are
>>> notified if degraded performance is observed. You can find proposal
>>> document
>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>> welcome you to share your thoughts on performance tests in general as well
>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>
>>>
>> --
Got feedback? go/pabloem-feedback
Re: Performance tests status and anomaly detection proposal
Posted by Chamikara Jayalath <ch...@google.com>.
Thanks Dariusz. This sounds great. Added some comments.
Also, +Jeff Gardner <ga...@google.com> who has experience on
performance regression analysis of integration tests.
Thanks,
Cham
On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
> @Etienne +1 to doing that! :) if we have both results (Nexmark and IOITs)
> in BQ we could use the same (similar?) tools to detect anomalies captured
> by Nexmark (if there's need for doing that).
>
> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
>
>> Very nice to see the dashboards !
>>
>> Regarding Kenn's comment: Nexmark supports outputing the results to
>> Bigquery so it could be easily integrated into the dashboards. We're, with
>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>> dashboard tables ?
>> WDYT?
>>
>> Etienne
>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>
>> This is very cool. So is it easy for someone to integrate the proposal to
>> regularly run Nexmark benchmarks and get those on the dashboard? (or a
>> separate one to keep IOs in their own page)
>>
>> Kenn
>>
>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>> dariusz.aniszewski@polidea.com> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Hello Beam devs!As you might already noticed, together with Łukasz
>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>> we’re working on adding some performance tests to the project. We were
>> following directions from the Testing I/O Transforms in Apache Beam
>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>> update in near future).We started from testing various FileBasedIOs as part
>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
>> for TFRecordIO, see BEAM-3945
>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
>> all the tests are maven-based, we responded to ongoing Gradle migration and
>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>> them via Gradle.All of those tests are executed on daily basis on Apache
>> Jenkins <https://builds.apache.org/> and their results are published to
>> individual BigQuery tables. There is also a dashboard on which tests
>> results may be viewed and
>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>> we have some amount of tests already, we’re currently working on a tool
>> that will analyze the results and search for anomalies, so devs are
>> notified if degraded performance is observed. You can find proposal
>> document
>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>> welcome you to share your thoughts on performance tests in general as well
>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>
>>
>
Re: Performance tests status and anomaly detection proposal
Posted by Łukasz Gajowy <lu...@gmail.com>.
@Etienne +1 to doing that! :) if we have both results (Nexmark and IOITs)
in BQ we could use the same (similar?) tools to detect anomalies captured
by Nexmark (if there's need for doing that).
2018-04-16 11:17 GMT+02:00 Etienne Chauchot <ec...@apache.org>:
> Very nice to see the dashboards !
>
> Regarding Kenn's comment: Nexmark supports outputing the results to
> Bigquery so it could be easily integrated into the dashboards. We're, with
> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
> dashboard tables ?
> WDYT?
>
> Etienne
> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>
> This is very cool. So is it easy for someone to integrate the proposal to
> regularly run Nexmark benchmarks and get those on the dashboard? (or a
> separate one to keep IOs in their own page)
>
> Kenn
>
> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
> dariusz.aniszewski@polidea.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Hello Beam devs!As you might already noticed, together with Łukasz
> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
> we’re working on adding some performance tests to the project. We were
> following directions from the Testing I/O Transforms in Apache Beam
> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
> update in near future).We started from testing various FileBasedIOs as part
> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
> for TFRecordIO, see BEAM-3945
> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
> was there, but was disabled; we fixed it and enabled)- HCatalogIO
> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
> all the tests are maven-based, we responded to ongoing Gradle migration and
> created PR <https://github.com/apache/beam/pull/5003> that allows running
> them via Gradle.All of those tests are executed on daily basis on Apache
> Jenkins <https://builds.apache.org/> and their results are published to
> individual BigQuery tables. There is also a dashboard on which tests
> results may be viewed and
> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
> we have some amount of tests already, we’re currently working on a tool
> that will analyze the results and search for anomalies, so devs are
> notified if degraded performance is observed. You can find proposal
> document
> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
> welcome you to share your thoughts on performance tests in general as well
> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>
>
Re: Performance tests status and anomaly detection proposal
Posted by Etienne Chauchot <ec...@apache.org>.
Very nice to see the dashboards !
Regarding Kenn's comment: Nexmark supports outputing the results to Bigquery so it could be easily integrated into the
dashboards. We're, with Kenn, scheduling Nexmark runs. We could configure the output to bigquery dashboard tables ?
WDYT?
Etienne
Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
> This is very cool. So is it easy for someone to integrate the proposal to regularly run Nexmark benchmarks and get
> those on the dashboard? (or a separate one to keep IOs in their own page)
>
> Kenn
>
> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <da...@polidea.com> wrote:
> > Hello Beam devs!
> >
> > As you might already noticed, together with Łukasz Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly
> > cc’d here) we’re working on adding some performance tests to the project. We were following directions from the
> > Testing I/O Transforms in Apache Beam site (which we plan to update in near future).
> >
> > We started from testing various FileBasedIOs as part of BEAM-3060. So far we have tests for:
> > TextIO (with and without compression)
> > AvroIO
> > XmlIO
> > TFRecordIO
> > that may run on following filesystems:
> > local
> > GCS
> > HDFS (except for TFRecordIO, see BEAM-3945)
> >
> > Besides FileBasedIOs we also covered:
> > HadoopInputFormatIO
> > MongoDBIO
> > JdbcIO (in this case test was there, but was disabled; we fixed it and enabled)
> > HCatalogIO (currently in PR)
> >
> > While currently all the tests are maven-based, we responded to ongoing Gradle migration and created PR that allows
> > running them via Gradle.
> >
> > All of those tests are executed on daily basis on Apache Jenkins and their results are published to individual
> > BigQuery tables. There is also a dashboard on which tests results may be viewed and compared:
> > https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
> >
> >
> > As we have some amount of tests already, we’re currently working on a tool that will analyze the results and search
> > for anomalies, so devs are notified if degraded performance is observed. You can find proposal document here:
> > https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
> >
> > We welcome you to share your thoughts on performance tests in general as well as proposed solution for anomaly
> > detection.
> >
> > Best,
> > Dariusz Aniszewski
> >
> >
> >
Re: Performance tests status and anomaly detection proposal
Posted by Kenneth Knowles <kl...@google.com>.
This is very cool. So is it easy for someone to integrate the proposal to
regularly run Nexmark benchmarks and get those on the dashboard? (or a
separate one to keep IOs in their own page)
Kenn
On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
dariusz.aniszewski@polidea.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Hello Beam devs!As you might already noticed, together with Łukasz
> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
> we’re working on adding some performance tests to the project. We were
> following directions from the Testing I/O Transforms in Apache Beam
> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
> update in near future).We started from testing various FileBasedIOs as part
> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except
> for TFRecordIO, see BEAM-3945
> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
> was there, but was disabled; we fixed it and enabled)- HCatalogIO
> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently
> all the tests are maven-based, we responded to ongoing Gradle migration and
> created PR <https://github.com/apache/beam/pull/5003> that allows running
> them via Gradle.All of those tests are executed on daily basis on Apache
> Jenkins <https://builds.apache.org/> and their results are published to
> individual BigQuery tables. There is also a dashboard on which tests
> results may be viewed and
> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
> we have some amount of tests already, we’re currently working on a tool
> that will analyze the results and search for anomalies, so devs are
> notified if degraded performance is observed. You can find proposal
> document
> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
> welcome you to share your thoughts on performance tests in general as well
> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>