You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Maximilian Michels <mx...@apache.org> on 2020/08/03 16:40:21 UTC
Re: Monitoring performance for releases
Here a first version of the updated release guide:
https://github.com/apache/beam/pull/12455
Feel free to comment.
-Max
On 29.07.20 17:27, Maximilian Michels wrote:
> Thanks! I'm following up with this PR to display the Flink Pardo
> streaming data: https://github.com/apache/beam/pull/12408
>
> Streaming data appears to be missing for Dataflow. We can revise the
> Jenkins jobs to add those.
>
> -Max
>
> On 29.07.20 17:01, Tyson Hamilton wrote:
>> Max,
>>
>> The runner dimension are present when hovering over a particular
>> graph. For some more info, the load test configurations can be found
>> here [1]. I didn't get a chance to look into them but there are tests
>> for all the runners there, possibly not for every loadtest.
>>
>> [1]: https://github.com/apache/beam/tree/master/.test-infra/jenkins
>>
>> -Tyson
>>
>> On Wed, Jul 29, 2020 at 3:46 AM Maximilian Michels <mxm@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Looks like the permissions won't be necessary because backup data
>> gets
>> loaded into the local InfluxDb instance which makes writing queries
>> locally possible.
>>
>> On 29.07.20 12:21, Maximilian Michels wrote:
>> > Thanks Michał!
>> >
>> > It is a bit tricky to verify the exported query works if I don't
>> have
>> > access to the data stored in InfluxDb.
>> >
>> > ==> Could somebody give me permissions to max.michels@gmail.com
>> <ma...@gmail.com> for
>> > apache-beam-testing such that I can setup a ssh port-forwarding
>> from the
>> > InfluxDb pod to my machine? I do have access to see the pods but
>> that is
>> > not enough.
>> >
>> >> I think that the only test data is from Python streaming tests,
>> which
>> >> are not implemented right now (check out
>> >>
>>
>> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
>>
>>
>> >> )
>> >
>> > Additionally, there is an entire dimension missing: Runners. I'm
>> > assuming this data is for Dataflow?
>> >
>> > -Max
>> >
>> > On 29.07.20 11:55, Michał Walenia wrote:
>> >> Hi there,
>> >>
>> >> > Indeed the Python load test data appears to be missing:
>> >> >
>> >>
>>
>> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
>>
>>
>> >>
>> >>
>> >> I think that the only test data is from Python streaming tests,
>> which
>> >> are not implemented right now (check out
>> >>
>>
>> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
>>
>>
>> >> )
>> >>
>> >> As for updating the dashboards, the manual for doing this is
>> here:
>> >>
>>
>> https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-UpdatingDashboards
>>
>>
>> >>
>> >>
>> >> I hope this helps,
>> >>
>> >> Michal
>> >>
>> >> On Mon, Jul 27, 2020 at 4:31 PM Maximilian Michels
>> <mxm@apache.org <ma...@apache.org>
>> >> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>> >>
>> >> Indeed the Python load test data appears to be missing:
>> >>
>> >>
>>
>> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
>>
>>
>> >>
>> >>
>> >> How do we typically modify the dashboards?
>> >>
>> >> It looks like we need to edit this json file:
>> >>
>> >>
>>
>> https://github.com/apache/beam/blob/8d460db620d2ff1257b0e092218294df15b409a1/.test-infra/metrics/grafana/dashboards/perftests_metrics/ParDo_Load_Tests.json#L81
>>
>>
>> >>
>> >>
>> >> I found some documentation on the deployment:
>> >>
>> >>
>>
>> https://cwiki.apache.org/confluence/display/BEAM/Test+Results+Monitoring
>> >>
>> >>
>> >> +1 for alerting or weekly emails including performance
>> numbers for
>> >> fixed
>> >> intervals (1d, 1w, 1m, previous release).
>> >>
>> >> +1 for linking the dashboards in the release guide to allow
>> for a
>> >> comparison as part of the release process.
>> >>
>> >> As a first step, consolidating all the data seems like the
>> most
>> >> pressing
>> >> problem to solve.
>> >>
>> >> @Kamil I could need some advice regarding how to proceed
>> updating the
>> >> dashboards.
>> >>
>> >> -Max
>> >>
>> >> On 22.07.20 20:20, Robert Bradshaw wrote:
>> >> > On Tue, Jul 21, 2020 at 9:58 AM Thomas Weise
>> <thw@apache.org <ma...@apache.org>
>> >> <mailto:thw@apache.org <ma...@apache.org>>
>> >> > <mailto:thw@apache.org <ma...@apache.org>
>> <mailto:thw@apache.org <ma...@apache.org>>>> wrote:
>> >> >
>> >> > It appears that there is coverage missing in the
>> Grafana
>> >> dashboards
>> >> > (it could also be that I just don't find it).
>> >> >
>> >> > For example:
>> >> >
>> >>
>> >>
>>
>> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
>>
>>
>> >>
>> >> >
>> >> > The GBK and ParDo tests have a selection for {batch,
>> >> streaming} and
>> >> > SDK. No coverage for streaming and python? There is
>> also no
>> >> runner
>> >> > option currently.
>> >> >
>> >> > We have seen repeated regressions with streaming,
>> Python,
>> >> Flink. The
>> >> > test has been contributed. It would be great if the
>> results
>> >> can be
>> >> > covered as part of release verification.
>> >> >
>> >> >
>> >> > Even better would be if we can use these dashboards (plus
>> >> alerting or
>> >> > similar?) to find issues before release verification.
>> It's much
>> >> easier
>> >> > to fix things earlier.
>> >> >
>> >> >
>> >> > Thomas
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Jul 21, 2020 at 7:55 AM Kamil Wasilewski
>> >> > <kamil.wasilewski@polidea.com
>> <ma...@polidea.com>
>> >> <mailto:kamil.wasilewski@polidea.com
>> <ma...@polidea.com>>
>> >> <mailto:kamil.wasilewski@polidea.com
>> <ma...@polidea.com>
>> >> <mailto:kamil.wasilewski@polidea.com
>> <ma...@polidea.com>>>>
>> >> > wrote:
>> >> >
>> >> > The prerequisite is that we have all the
>> stats in one
>> >> place.
>> >> > They seem
>> >> > to be scattered across
>> >> http://metrics.beam.apache.org and
>> >> > https://apache-beam-testing.appspot.com.
>> >> >
>> >> > Would it be possible to consolidate the two,
>> i.e.
>> >> use the
>> >> > Grafana-based
>> >> > dashboard to load the legacy stats?
>> >> >
>> >> >
>> >> > I'm pretty sure that all dashboards have been
>> moved to
>> >> > http://metrics.beam.apache.org. Let me know if I missed
>> >> > something during the migration.
>> >> >
>> >> > I think we should turn off
>> >> > https://apache-beam-testing.appspot.com in the near
>> future. New
>> >> > Grafana-based dashboards have been working
>> seamlessly for
>> >> some
>> >> > time now and there's no point in maintaining the
>> older
>> >> solution.
>> >> > We'd also avoid ambiguity in where the stats
>> should be
>> >> looked for.
>> >> >
>> >> > Kamil
>> >> >
>> >> > On Tue, Jul 21, 2020 at 4:17 PM Maximilian Michels
>> >> > <mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> >> >
>> >> > > It doesn't support https. I had to add an
>> >> exception to
>> >> > the HTTPS Everywhere extension for
>> >> "metrics.beam.apache.org <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> > <http://metrics.beam.apache.org>".
>> >> >
>> >> > *facepalm* Thanks Udi! It would always hang
>> on me
>> >> because I
>> >> > use HTTPS
>> >> > Everywhere.
>> >> >
>> >> > > To be explicit, I am supporting the idea of
>> >> reviewing the
>> >> > release guide but not changing the release
>> process
>> >> for the
>> >> > already in-progress release.
>> >> >
>> >> > I consider the release guide immutable for the
>> >> process of a
>> >> > release.
>> >> > Thus, a change to the release guide can only
>> affect
>> >> new
>> >> > upcoming
>> >> > releases, not an in-process release.
>> >> >
>> >> > > +1 and I think we can also evaluate
>> whether flaky
>> >> tests
>> >> > should be reviewed as release blockers or
>> not. Some
>> >> flaky
>> >> > tests would be hiding real issues our users
>> could
>> >> face.
>> >> >
>> >> > Flaky tests are also worth to take into
>> account when
>> >> > releasing, but a
>> >> > little harder to find because may just happen
>> to pass
>> >> during
>> >> > building
>> >> > the release. It is possible though if we
>> strictly
>> >> capture
>> >> > flaky tests
>> >> > via JIRA and mark them with the Fix Version
>> for the
>> >> release.
>> >> >
>> >> > > We keep accumulating dashboards and
>> >> > > tests that few people care about, so it is
>> >> probably worth
>> >> > that we use
>> >> > > them or get a way to alert us of
>> regressions
>> >> during the
>> >> > release cycle
>> >> > > to catch this even before the RCs.
>> >> >
>> >> > +1 The release guide should be explicit about
>> which
>> >> > performance test
>> >> > results to evaluate.
>> >> >
>> >> > The prerequisite is that we have all the
>> stats in one
>> >> place.
>> >> > They seem
>> >> > to be scattered across
>> >> http://metrics.beam.apache.org and
>> >> > https://apache-beam-testing.appspot.com.
>> >> >
>> >> > Would it be possible to consolidate the two,
>> i.e.
>> >> use the
>> >> > Grafana-based
>> >> > dashboard to load the legacy stats?
>> >> >
>> >> > For the evaluation during the release
>> process, I
>> >> suggest to
>> >> > use a
>> >> > standardized set of performance tests for all
>> >> runners, e.g.:
>> >> >
>> >> > - Nexmark
>> >> > - ParDo (Classic/Portable)
>> >> > - GroupByKey
>> >> > - IO
>> >> >
>> >> >
>> >> > -Max
>> >> >
>> >> > On 21.07.20 01:23, Ahmet Altay wrote:
>> >> > >
>> >> > > On Mon, Jul 20, 2020 at 3:07 PM Ismaël
>> Mejía
>> >> > <iemejia@gmail.com <ma...@gmail.com>
>> <mailto:iemejia@gmail.com <ma...@gmail.com>>
>> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
>> <mailto:iemejia@gmail.com <ma...@gmail.com>>>
>> >> > > <mailto:iemejia@gmail.com
>> <ma...@gmail.com>
>> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>
>> <mailto:iemejia@gmail.com <ma...@gmail.com>
>> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>>>>
>> wrote:
>> >> > >
>> >> > > +1
>> >> > >
>> >> > > This is not in the release guide and
>> we should
>> >> > probably re evaluate if
>> >> > > this should be a release blocking
>> reason.
>> >> > > Of course exceptionally a performance
>> >> regression
>> >> > could be motivated by
>> >> > > a correctness fix or a worth refactor,
>> so we
>> >> should
>> >> > consider this.
>> >> > >
>> >> > >
>> >> > > +1 and I think we can also evaluate whether
>> >> flaky tests
>> >> > should be
>> >> > > reviewed as release blockers or not. Some
>> flaky
>> >> tests
>> >> > would be hiding
>> >> > > real issues our users could face.
>> >> > >
>> >> > > To be explicit, I am supporting the idea of
>> >> reviewing the
>> >> > release guide
>> >> > > but not changing the release process for
>> the
>> >> already
>> >> > in-progress release.
>> >> > >
>> >> > >
>> >> > > We have been tracking and fixing
>> performance
>> >> > regressions multiple
>> >> > > times found simply by checking the
>> nexmark
>> >> tests
>> >> > including on the
>> >> > > ongoing 2.23.0 release so value is
>> there.
>> >> Nexmark
>> >> > does not cover yet
>> >> > > python and portable runners so we are
>> probably
>> >> still
>> >> > missing many
>> >> > > issues and it is worth to work on
>> this. In any
>> >> case
>> >> > we should probably
>> >> > > decide what validations matter. We keep
>> >> accumulating
>> >> > dashboards and
>> >> > > tests that few people care about, so
>> it is
>> >> probably
>> >> > worth that we use
>> >> > > them or get a way to alert us of
>> regressions
>> >> during
>> >> > the release cycle
>> >> > > to catch this even before the RCs.
>> >> > >
>> >> > >
>> >> > > I agree. And if we cannot use
>> dashboards/tests in a
>> >> > meaningful way, IMO
>> >> > > we can remove them. There is not much
>> value to
>> >> maintain
>> >> > them if they do
>> >> > > not provide important signals.
>> >> > >
>> >> > >
>> >> > > On Fri, Jul 10, 2020 at 9:30 PM Udi
>> Meiri
>> >> > <ehudm@google.com <ma...@google.com>
>> <mailto:ehudm@google.com <ma...@google.com>>
>> >> <mailto:ehudm@google.com <ma...@google.com>
>> <mailto:ehudm@google.com <ma...@google.com>>>
>> >> > > <mailto:ehudm@google.com
>> <ma...@google.com>
>> >> <mailto:ehudm@google.com <ma...@google.com>>
>> <mailto:ehudm@google.com <ma...@google.com>
>> >> <mailto:ehudm@google.com <ma...@google.com>>>>>
>> >> > wrote:
>> >> > > >
>> >> > > > On Thu, Jul 9, 2020 at 12:48 PM
>> Maximilian
>> >> Michels
>> >> > > <mxm@apache.org
>> <ma...@apache.org> <mailto:mxm@apache.org
>> <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>
>> >> > <mailto:mxm@apache.org
>> <ma...@apache.org> <mailto:mxm@apache.org
>> <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
>> >> > > >>
>> >> > > >> Not yet, I just learned about the
>> >> migration to a
>> >> > new frontend,
>> >> > > including
>> >> > > >> a new backend (InfluxDB instead of
>> >> BigQuery).
>> >> > > >>
>> >> > > >> > - Are the metrics available on
>> >> > metrics.beam.apache.org <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> <http://metrics.beam.apache.org>
>> >> > > <http://metrics.beam.apache.org>?
>> >> > > >>
>> >> > > >> Is http://metrics.beam.apache.org
>> online?
>> >> I was
>> >> > never able to
>> >> > > access it.
>> >> > > >
>> >> > > >
>> >> > > > It doesn't support https. I had to
>> add an
>> >> > exception to the HTTPS
>> >> > > Everywhere extension for
>> >> "metrics.beam.apache.org <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> > <http://metrics.beam.apache.org>
>> >> > > <http://metrics.beam.apache.org>".
>> >> > > >
>> >> > > >>
>> >> > > >>
>> >> > > >> > - What is the feature delta
>> between
>> >> usinig
>> >> > > metrics.beam.apache.org
>> <http://metrics.beam.apache.org>
>> >> <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> > <http://metrics.beam.apache.org> (much
>> >> > > better UI) and using
>> >> apache-beam-testing.appspot.com
>> <http://apache-beam-testing.appspot.com>
>> >> <http://apache-beam-testing.appspot.com>
>> >> > <http://apache-beam-testing.appspot.com>
>> >> > >
>> <http://apache-beam-testing.appspot.com>?
>> >> > > >>
>> >> > > >> AFAIK it is an ongoing migration
>> and the
>> >> delta
>> >> > appears to be high.
>> >> > > >>
>> >> > > >> > - Can we notice regressions
>> faster than
>> >> > release cadence?
>> >> > > >>
>> >> > > >> Absolutely! A report with the
>> latest
>> >> numbers
>> >> > including
>> >> > > statistics about
>> >> > > >> the growth of metrics would be
>> useful.
>> >> > > >>
>> >> > > >> > - Can we get automated alerts?
>> >> > > >>
>> >> > > >> I think we could setup a Jenkins
>> job to do
>> >> this.
>> >> > > >>
>> >> > > >> -Max
>> >> > > >>
>> >> > > >> On 09.07.20 20:26, Kenneth Knowles
>> wrote:
>> >> > > >> > Questions:
>> >> > > >> >
>> >> > > >> > - Are the metrics available on
>> >> > metrics.beam.apache.org <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> <http://metrics.beam.apache.org>
>> >> > > <http://metrics.beam.apache.org>
>> >> > > >> > <http://metrics.beam.apache.org>?
>> >> > > >> > - What is the feature delta
>> between
>> >> usinig
>> >> > > metrics.beam.apache.org
>> <http://metrics.beam.apache.org>
>> >> <http://metrics.beam.apache.org>
>> <http://metrics.beam.apache.org>
>> >> > <http://metrics.beam.apache.org>
>> >> > > >> > <http://metrics.beam.apache.org>
>> (much
>> >> better
>> >> > UI) and using
>> >> > > >> > apache-beam-testing.appspot.com
>> <http://apache-beam-testing.appspot.com>
>> >> <http://apache-beam-testing.appspot.com>
>> >> > <http://apache-beam-testing.appspot.com>
>> >> > >
>> <http://apache-beam-testing.appspot.com>
>> >> > >
>> <http://apache-beam-testing.appspot.com>?
>> >> > > >> > - Can we notice regressions
>> faster than
>> >> > release cadence?
>> >> > > >> > - Can we get automated alerts?
>> >> > > >> >
>> >> > > >> > Kenn
>> >> > > >> >
>> >> > > >> > On Thu, Jul 9, 2020 at 10:21 AM
>> >> Maximilian Michels
>> >> > > <mxm@apache.org
>> <ma...@apache.org> <mailto:mxm@apache.org
>> <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>
>> >> > <mailto:mxm@apache.org
>> <ma...@apache.org> <mailto:mxm@apache.org
>> <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>>
>> >> > > >> > <mailto:mxm@apache.org
>> <ma...@apache.org>
>> >> <mailto:mxm@apache.org <ma...@apache.org>>
>> <mailto:mxm@apache.org <ma...@apache.org>
>> >> <mailto:mxm@apache.org <ma...@apache.org>>>
>> >> > <mailto:mxm@apache.org
>> <ma...@apache.org> <mailto:mxm@apache.org
>> <ma...@apache.org>>
>> >> <mailto:mxm@apache.org <ma...@apache.org>
>> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
>> >> > > >> >
>> >> > > >> > Hi,
>> >> > > >> >
>> >> > > >> > We recently saw an
>> increase in
>> >> latency
>> >> > migrating from Beam
>> >> > > 2.18.0 to
>> >> > > >> > 2.21.0 (Python SDK with Flink
>> >> Runner). This
>> >> > proofed very
>> >> > > hard to debug
>> >> > > >> > and it looks like each
>> version in
>> >> between
>> >> > the two versions
>> >> > > let to
>> >> > > >> > increased latency.
>> >> > > >> >
>> >> > > >> > This is not the first time
>> we saw
>> >> issues
>> >> > when migrating,
>> >> > > another
>> >> > > >> > time we
>> >> > > >> > had a decline in
>> checkpointing
>> >> performance
>> >> > and thus added a
>> >> > > >> > checkpointing test [1] and
>> dashboard
>> >> [2] (see
>> >> > > checkpointing widget).
>> >> > > >> >
>> >> > > >> > That makes me wonder if we
>> should
>> >> monitor
>> >> > performance
>> >> > > (throughput /
>> >> > > >> > latency) for basic use cases
>> as part
>> >> of the
>> >> > release
>> >> > > testing. Currently,
>> >> > > >> > our release guide [3]
>> mentions
>> >> running
>> >> > examples but not
>> >> > > evaluating the
>> >> > > >> > performance. I think it
>> would be good
>> >> > practice to check
>> >> > > relevant charts
>> >> > > >> > with performance
>> measurements as
>> >> part of of
>> >> > the release
>> >> > > process. The
>> >> > > >> > release guide should reflect
>> that.
>> >> > > >> >
>> >> > > >> > WDYT?
>> >> > > >> >
>> >> > > >> > -Max
>> >> > > >> >
>> >> > > >> > PS: Of course, this requires
>> tests
>> >> and
>> >> > metrics to be
>> >> > > available. This PR
>> >> > > >> > adds latency measurements
>> to the
>> >> load tests
>> >> > [4].
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > [1]
>> >> https://github.com/apache/beam/pull/11558
>> >> > > >> > [2]
>> >> > > >> >
>> >> > >
>> >> >
>> >>
>> >>
>>
>> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
>>
>>
>> >>
>> >> > > >> > [3]
>> >> > https://beam.apache.org/contribute/release-guide/
>> >> > > >> > [4]
>> >> https://github.com/apache/beam/pull/12065
>> >> > > >> >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Michał Walenia
>> >> Polidea <https://www.polidea.com/> | Software Engineer
>> >>
>> >> M: +48 791 432 002 <tel:+48%20791%20432%20002> <tel:+48791432002
>> <tel:+48%20791%20432%20002>>
>> >> E: michal.walenia@polidea.com
>> <ma...@polidea.com>
>> <mailto:michal.walenia@polidea.com
>> <ma...@polidea.com>>
>> >>
>> >> Unique Tech
>> >> Check out our projects! <https://www.polidea.com/our-work>
>> >>
>>
Re: Monitoring performance for releases
Posted by Tobiasz Kędzierski <to...@polidea.com>.
Hi,
I created PR[1] that adds a weekly email with statistics report which
compares mean values from current and previous week, based on InfluxDB. It
has links to corresponding grafana dashboards as well.
Preview of generated mail is available [2]. Please take into account that I
used a few days old data for generating this preview so values may not
reflect state in the grafana dashboards.
Looking forward to your reviews.
[1] https://github.com/apache/beam/pull/12685
[2]
https://storage.googleapis.com/beam-wheel-test-bucket-uscentral1/metrics-report/beam-metrics_report.html
BR Tobiasz
On Thu, Aug 6, 2020 at 6:28 PM Maximilian Michels <mx...@apache.org> wrote:
> Robert, this is not too far off what I'm proposing. We can always create
> JIRA issues for performance regressions and mark them with a Fix
> Version. Especially, the time of the release is a good time to
> re-evaluate whether some gross performance regressions can be detected.
> Of course, if it's a more gradual and less noticeable once, we might
> miss it.
>
> I agree that good performance is a continuous effort, but many times
> there are actual problems which can be dealt with at the time of the
> release. Those might be very hard to fix a couple of releases down the
> line because they are layered by new problems and much harder to find.
>
> -Max
>
> On 03.08.20 22:17, Robert Bradshaw wrote:
> > I have to admit I still have some qualms about tying detecting and
> > fixing performance regressions as part of the release process (which is
> > onerous enough as it is). Instead, I think we'd be better off with a
> > separate process to detect and triage performance issues, which, when
> > they occur, may merit filing a blocker which will require fixing before
> > the release just like any other blocker would. Hopefully this would
> > result in issues being detected (and resolved) sooner.
> >
> > That being said, if a release is known to have performance regressions,
> > that should be called out when the RCs are cut, and if not resolved,
> > probably as part of the release notes as well.
> >
> > On Mon, Aug 3, 2020 at 9:40 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Here a first version of the updated release guide:
> > https://github.com/apache/beam/pull/12455
> >
> > Feel free to comment.
> >
> > -Max
> >
> > On 29.07.20 17:27, Maximilian Michels wrote:
> > > Thanks! I'm following up with this PR to display the Flink Pardo
> > > streaming data: https://github.com/apache/beam/pull/12408
> > >
> > > Streaming data appears to be missing for Dataflow. We can revise
> the
> > > Jenkins jobs to add those.
> > >
> > > -Max
> > >
> > > On 29.07.20 17:01, Tyson Hamilton wrote:
> > >> Max,
> > >>
> > >> The runner dimension are present when hovering over a particular
> > >> graph. For some more info, the load test configurations can be
> > found
> > >> here [1]. I didn't get a chance to look into them but there are
> > tests
> > >> for all the runners there, possibly not for every loadtest.
> > >>
> > >> [1]:
> https://github.com/apache/beam/tree/master/.test-infra/jenkins
> > >>
> > >> -Tyson
> > >>
> > >> On Wed, Jul 29, 2020 at 3:46 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >>
> > >> Looks like the permissions won't be necessary because backup
> > data
> > >> gets
> > >> loaded into the local InfluxDb instance which makes writing
> > queries
> > >> locally possible.
> > >>
> > >> On 29.07.20 12:21, Maximilian Michels wrote:
> > >> > Thanks Michał!
> > >> >
> > >> > It is a bit tricky to verify the exported query works if
> > I don't
> > >> have
> > >> > access to the data stored in InfluxDb.
> > >> >
> > >> > ==> Could somebody give me permissions to
> > max.michels@gmail.com <ma...@gmail.com>
> > >> <mailto:max.michels@gmail.com
> > <ma...@gmail.com>> for
> > >> > apache-beam-testing such that I can setup a ssh
> > port-forwarding
> > >> from the
> > >> > InfluxDb pod to my machine? I do have access to see the
> > pods but
> > >> that is
> > >> > not enough.
> > >> >
> > >> >> I think that the only test data is from Python streaming
> > tests,
> > >> which
> > >> >> are not implemented right now (check out
> > >> >>
> > >>
> > >>
> >
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
> >
> > >>
> > >>
> > >> >> )
> > >> >
> > >> > Additionally, there is an entire dimension missing:
> > Runners. I'm
> > >> > assuming this data is for Dataflow?
> > >> >
> > >> > -Max
> > >> >
> > >> > On 29.07.20 11:55, Michał Walenia wrote:
> > >> >> Hi there,
> > >> >>
> > >> >> > Indeed the Python load test data appears to be
> missing:
> > >> >> >
> > >> >>
> > >>
> > >>
> >
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
> >
> > >>
> > >>
> > >> >>
> > >> >>
> > >> >> I think that the only test data is from Python streaming
> > tests,
> > >> which
> > >> >> are not implemented right now (check out
> > >> >>
> > >>
> > >>
> >
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
> >
> > >>
> > >>
> > >> >> )
> > >> >>
> > >> >> As for updating the dashboards, the manual for doing
> > this is
> > >> here:
> > >> >>
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-UpdatingDashboards
> >
> > >>
> > >>
> > >> >>
> > >> >>
> > >> >> I hope this helps,
> > >> >>
> > >> >> Michal
> > >> >>
> > >> >> On Mon, Jul 27, 2020 at 4:31 PM Maximilian Michels
> > >> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >> >>
> > >> >> Indeed the Python load test data appears to be
> missing:
> > >> >>
> > >> >>
> > >>
> > >>
> >
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
> >
> > >>
> > >>
> > >> >>
> > >> >>
> > >> >> How do we typically modify the dashboards?
> > >> >>
> > >> >> It looks like we need to edit this json file:
> > >> >>
> > >> >>
> > >>
> > >>
> >
> https://github.com/apache/beam/blob/8d460db620d2ff1257b0e092218294df15b409a1/.test-infra/metrics/grafana/dashboards/perftests_metrics/ParDo_Load_Tests.json#L81
> >
> > >>
> > >>
> > >> >>
> > >> >>
> > >> >> I found some documentation on the deployment:
> > >> >>
> > >> >>
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/BEAM/Test+Results+Monitoring
> > >> >>
> > >> >>
> > >> >> +1 for alerting or weekly emails including
> performance
> > >> numbers for
> > >> >> fixed
> > >> >> intervals (1d, 1w, 1m, previous release).
> > >> >>
> > >> >> +1 for linking the dashboards in the release guide
> > to allow
> > >> for a
> > >> >> comparison as part of the release process.
> > >> >>
> > >> >> As a first step, consolidating all the data seems
> > like the
> > >> most
> > >> >> pressing
> > >> >> problem to solve.
> > >> >>
> > >> >> @Kamil I could need some advice regarding how to
> proceed
> > >> updating the
> > >> >> dashboards.
> > >> >>
> > >> >> -Max
> > >> >>
> > >> >> On 22.07.20 20:20, Robert Bradshaw wrote:
> > >> >> > On Tue, Jul 21, 2020 at 9:58 AM Thomas Weise
> > >> <thw@apache.org <ma...@apache.org>
> > <mailto:thw@apache.org <ma...@apache.org>>
> > >> >> <mailto:thw@apache.org <ma...@apache.org>
> > <mailto:thw@apache.org <ma...@apache.org>>>
> > >> >> > <mailto:thw@apache.org <ma...@apache.org>
> > <mailto:thw@apache.org <ma...@apache.org>>
> > >> <mailto:thw@apache.org <ma...@apache.org>
> > <mailto:thw@apache.org <ma...@apache.org>>>>> wrote:
> > >> >> >
> > >> >> > It appears that there is coverage missing in
> the
> > >> Grafana
> > >> >> dashboards
> > >> >> > (it could also be that I just don't find it).
> > >> >> >
> > >> >> > For example:
> > >> >> >
> > >> >>
> > >> >>
> > >>
> > >>
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
> >
> > >>
> > >>
> > >> >>
> > >> >> >
> > >> >> > The GBK and ParDo tests have a selection for
> > {batch,
> > >> >> streaming} and
> > >> >> > SDK. No coverage for streaming and python?
> > There is
> > >> also no
> > >> >> runner
> > >> >> > option currently.
> > >> >> >
> > >> >> > We have seen repeated regressions with
> > streaming,
> > >> Python,
> > >> >> Flink. The
> > >> >> > test has been contributed. It would be great
> > if the
> > >> results
> > >> >> can be
> > >> >> > covered as part of release verification.
> > >> >> >
> > >> >> >
> > >> >> > Even better would be if we can use these
> > dashboards (plus
> > >> >> alerting or
> > >> >> > similar?) to find issues before release
> verification.
> > >> It's much
> > >> >> easier
> > >> >> > to fix things earlier.
> > >> >> >
> > >> >> >
> > >> >> > Thomas
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Tue, Jul 21, 2020 at 7:55 AM Kamil
> Wasilewski
> > >> >> > <kamil.wasilewski@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>>
> > >> >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>>>
> > >> >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>>
> > >> >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:kamil.wasilewski@polidea.com
> > <ma...@polidea.com>>>>>
> > >> >> > wrote:
> > >> >> >
> > >> >> > The prerequisite is that we have all
> the
> > >> stats in one
> > >> >> place.
> > >> >> > They seem
> > >> >> > to be scattered across
> > >> >> http://metrics.beam.apache.org and
> > >> >> > https://apache-beam-testing.appspot.com.
> > >> >> >
> > >> >> > Would it be possible to consolidate
> > the two,
> > >> i.e.
> > >> >> use the
> > >> >> > Grafana-based
> > >> >> > dashboard to load the legacy stats?
> > >> >> >
> > >> >> >
> > >> >> > I'm pretty sure that all dashboards have
> been
> > >> moved to
> > >> >> > http://metrics.beam.apache.org. Let me know if I
> > missed
> > >> >> > something during the migration.
> > >> >> >
> > >> >> > I think we should turn off
> > >> >> > https://apache-beam-testing.appspot.com in the
> near
> > >> future. New
> > >> >> > Grafana-based dashboards have been working
> > >> seamlessly for
> > >> >> some
> > >> >> > time now and there's no point in
> > maintaining the
> > >> older
> > >> >> solution.
> > >> >> > We'd also avoid ambiguity in where the
> stats
> > >> should be
> > >> >> looked for.
> > >> >> >
> > >> >> > Kamil
> > >> >> >
> > >> >> > On Tue, Jul 21, 2020 at 4:17 PM
> > Maximilian Michels
> > >> >> > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > >> >> >
> > >> >> > > It doesn't support https. I had to
> > add an
> > >> >> exception to
> > >> >> > the HTTPS Everywhere extension for
> > >> >> "metrics.beam.apache.org
> > <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> > <http://metrics.beam.apache.org>".
> > >> >> >
> > >> >> > *facepalm* Thanks Udi! It would
> > always hang
> > >> on me
> > >> >> because I
> > >> >> > use HTTPS
> > >> >> > Everywhere.
> > >> >> >
> > >> >> > > To be explicit, I am supporting
> > the idea of
> > >> >> reviewing the
> > >> >> > release guide but not changing the
> > release
> > >> process
> > >> >> for the
> > >> >> > already in-progress release.
> > >> >> >
> > >> >> > I consider the release guide
> > immutable for the
> > >> >> process of a
> > >> >> > release.
> > >> >> > Thus, a change to the release guide
> > can only
> > >> affect
> > >> >> new
> > >> >> > upcoming
> > >> >> > releases, not an in-process release.
> > >> >> >
> > >> >> > > +1 and I think we can also evaluate
> > >> whether flaky
> > >> >> tests
> > >> >> > should be reviewed as release
> blockers or
> > >> not. Some
> > >> >> flaky
> > >> >> > tests would be hiding real issues our
> > users
> > >> could
> > >> >> face.
> > >> >> >
> > >> >> > Flaky tests are also worth to take
> into
> > >> account when
> > >> >> > releasing, but a
> > >> >> > little harder to find because may
> > just happen
> > >> to pass
> > >> >> during
> > >> >> > building
> > >> >> > the release. It is possible though if
> we
> > >> strictly
> > >> >> capture
> > >> >> > flaky tests
> > >> >> > via JIRA and mark them with the Fix
> > Version
> > >> for the
> > >> >> release.
> > >> >> >
> > >> >> > > We keep accumulating dashboards and
> > >> >> > > tests that few people care about,
> > so it is
> > >> >> probably worth
> > >> >> > that we use
> > >> >> > > them or get a way to alert us of
> > >> regressions
> > >> >> during the
> > >> >> > release cycle
> > >> >> > > to catch this even before the RCs.
> > >> >> >
> > >> >> > +1 The release guide should be
> > explicit about
> > >> which
> > >> >> > performance test
> > >> >> > results to evaluate.
> > >> >> >
> > >> >> > The prerequisite is that we have all
> the
> > >> stats in one
> > >> >> place.
> > >> >> > They seem
> > >> >> > to be scattered across
> > >> >> http://metrics.beam.apache.org and
> > >> >> > https://apache-beam-testing.appspot.com.
> > >> >> >
> > >> >> > Would it be possible to consolidate
> > the two,
> > >> i.e.
> > >> >> use the
> > >> >> > Grafana-based
> > >> >> > dashboard to load the legacy stats?
> > >> >> >
> > >> >> > For the evaluation during the release
> > >> process, I
> > >> >> suggest to
> > >> >> > use a
> > >> >> > standardized set of performance tests
> > for all
> > >> >> runners, e.g.:
> > >> >> >
> > >> >> > - Nexmark
> > >> >> > - ParDo (Classic/Portable)
> > >> >> > - GroupByKey
> > >> >> > - IO
> > >> >> >
> > >> >> >
> > >> >> > -Max
> > >> >> >
> > >> >> > On 21.07.20 01:23, Ahmet Altay wrote:
> > >> >> > >
> > >> >> > > On Mon, Jul 20, 2020 at 3:07 PM
> > Ismaël
> > >> Mejía
> > >> >> > <iemejia@gmail.com
> > <ma...@gmail.com> <mailto:iemejia@gmail.com
> > <ma...@gmail.com>>
> > >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> > >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>>>
> > >> >> > > <mailto:iemejia@gmail.com
> > <ma...@gmail.com>
> > >> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> > >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>>>>>
> > >> wrote:
> > >> >> > >
> > >> >> > > +1
> > >> >> > >
> > >> >> > > This is not in the release
> > guide and
> > >> we should
> > >> >> > probably re evaluate if
> > >> >> > > this should be a release
> blocking
> > >> reason.
> > >> >> > > Of course exceptionally a
> > performance
> > >> >> regression
> > >> >> > could be motivated by
> > >> >> > > a correctness fix or a worth
> > refactor,
> > >> so we
> > >> >> should
> > >> >> > consider this.
> > >> >> > >
> > >> >> > >
> > >> >> > > +1 and I think we can also
> > evaluate whether
> > >> >> flaky tests
> > >> >> > should be
> > >> >> > > reviewed as release blockers or
> > not. Some
> > >> flaky
> > >> >> tests
> > >> >> > would be hiding
> > >> >> > > real issues our users could face.
> > >> >> > >
> > >> >> > > To be explicit, I am supporting
> > the idea of
> > >> >> reviewing the
> > >> >> > release guide
> > >> >> > > but not changing the release
> > process for
> > >> the
> > >> >> already
> > >> >> > in-progress release.
> > >> >> > >
> > >> >> > >
> > >> >> > > We have been tracking and
> fixing
> > >> performance
> > >> >> > regressions multiple
> > >> >> > > times found simply by checking
> the
> > >> nexmark
> > >> >> tests
> > >> >> > including on the
> > >> >> > > ongoing 2.23.0 release so
> > value is
> > >> there.
> > >> >> Nexmark
> > >> >> > does not cover yet
> > >> >> > > python and portable runners so
> > we are
> > >> probably
> > >> >> still
> > >> >> > missing many
> > >> >> > > issues and it is worth to work
> on
> > >> this. In any
> > >> >> case
> > >> >> > we should probably
> > >> >> > > decide what validations
> > matter. We keep
> > >> >> accumulating
> > >> >> > dashboards and
> > >> >> > > tests that few people care
> > about, so
> > >> it is
> > >> >> probably
> > >> >> > worth that we use
> > >> >> > > them or get a way to alert us
> of
> > >> regressions
> > >> >> during
> > >> >> > the release cycle
> > >> >> > > to catch this even before the
> RCs.
> > >> >> > >
> > >> >> > >
> > >> >> > > I agree. And if we cannot use
> > >> dashboards/tests in a
> > >> >> > meaningful way, IMO
> > >> >> > > we can remove them. There is not
> much
> > >> value to
> > >> >> maintain
> > >> >> > them if they do
> > >> >> > > not provide important signals.
> > >> >> > >
> > >> >> > >
> > >> >> > > On Fri, Jul 10, 2020 at 9:30
> > PM Udi
> > >> Meiri
> > >> >> > <ehudm@google.com
> > <ma...@google.com> <mailto:ehudm@google.com
> > <ma...@google.com>>
> > >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>>
> > >> >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>
> > >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>>>
> > >> >> > > <mailto:ehudm@google.com
> > <ma...@google.com>
> > >> <mailto:ehudm@google.com <ma...@google.com>>
> > >> >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>>
> > >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>
> > >> >> <mailto:ehudm@google.com <ma...@google.com>
> > <mailto:ehudm@google.com <ma...@google.com>>>>>>
> > >> >> > wrote:
> > >> >> > > >
> > >> >> > > > On Thu, Jul 9, 2020 at
> 12:48 PM
> > >> Maximilian
> > >> >> Michels
> > >> >> > > <mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > >> >> > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > >> >> > > >>
> > >> >> > > >> Not yet, I just learned
> > about the
> > >> >> migration to a
> > >> >> > new frontend,
> > >> >> > > including
> > >> >> > > >> a new backend (InfluxDB
> > instead of
> > >> >> BigQuery).
> > >> >> > > >>
> > >> >> > > >> > - Are the metrics
> > available on
> > >> >> > metrics.beam.apache.org
> > <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> <http://metrics.beam.apache.org>
> > >> >> > > <
> http://metrics.beam.apache.org>?
> > >> >> > > >>
> > >> >> > > >> Is
> > http://metrics.beam.apache.org
> > >> online?
> > >> >> I was
> > >> >> > never able to
> > >> >> > > access it.
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > It doesn't support https. I
> > had to
> > >> add an
> > >> >> > exception to the HTTPS
> > >> >> > > Everywhere extension for
> > >> >> "metrics.beam.apache.org
> > <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> > <http://metrics.beam.apache.org>
> > >> >> > > <
> http://metrics.beam.apache.org>".
> > >> >> > > >
> > >> >> > > >>
> > >> >> > > >>
> > >> >> > > >> > - What is the feature
> delta
> > >> between
> > >> >> usinig
> > >> >> > > metrics.beam.apache.org
> > <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> > <http://metrics.beam.apache.org>
> (much
> > >> >> > > better UI) and using
> > >> >> apache-beam-testing.appspot.com
> > <http://apache-beam-testing.appspot.com>
> > >> <http://apache-beam-testing.appspot.com>
> > >> >> <http://apache-beam-testing.appspot.com>
> > >> >> > <
> http://apache-beam-testing.appspot.com>
> > >> >> > >
> > >> <http://apache-beam-testing.appspot.com>?
> > >> >> > > >>
> > >> >> > > >> AFAIK it is an ongoing
> > migration
> > >> and the
> > >> >> delta
> > >> >> > appears to be high.
> > >> >> > > >>
> > >> >> > > >> > - Can we notice
> regressions
> > >> faster than
> > >> >> > release cadence?
> > >> >> > > >>
> > >> >> > > >> Absolutely! A report with
> the
> > >> latest
> > >> >> numbers
> > >> >> > including
> > >> >> > > statistics about
> > >> >> > > >> the growth of metrics
> > would be
> > >> useful.
> > >> >> > > >>
> > >> >> > > >> > - Can we get automated
> > alerts?
> > >> >> > > >>
> > >> >> > > >> I think we could setup a
> > Jenkins
> > >> job to do
> > >> >> this.
> > >> >> > > >>
> > >> >> > > >> -Max
> > >> >> > > >>
> > >> >> > > >> On 09.07.20 20:26, Kenneth
> > Knowles
> > >> wrote:
> > >> >> > > >> > Questions:
> > >> >> > > >> >
> > >> >> > > >> > - Are the metrics
> > available on
> > >> >> > metrics.beam.apache.org
> > <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> <http://metrics.beam.apache.org>
> > >> >> > > <
> http://metrics.beam.apache.org>
> > >> >> > > >> >
> > <http://metrics.beam.apache.org>?
> > >> >> > > >> > - What is the feature
> > delta
> > >> between
> > >> >> usinig
> > >> >> > > metrics.beam.apache.org
> > <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> <http://metrics.beam.apache.org>
> > >> <http://metrics.beam.apache.org>
> > >> >> > <http://metrics.beam.apache.org>
> > >> >> > > >> >
> > <http://metrics.beam.apache.org>
> > >> (much
> > >> >> better
> > >> >> > UI) and using
> > >> >> > > >> >
> > apache-beam-testing.appspot.com <
> http://apache-beam-testing.appspot.com>
> > >> <http://apache-beam-testing.appspot.com>
> > >> >> <http://apache-beam-testing.appspot.com>
> > >> >> > <
> http://apache-beam-testing.appspot.com>
> > >> >> > >
> > >> <http://apache-beam-testing.appspot.com>
> > >> >> > >
> > >> <http://apache-beam-testing.appspot.com>?
> > >> >> > > >> > - Can we notice
> > regressions
> > >> faster than
> > >> >> > release cadence?
> > >> >> > > >> > - Can we get automated
> > alerts?
> > >> >> > > >> >
> > >> >> > > >> > Kenn
> > >> >> > > >> >
> > >> >> > > >> > On Thu, Jul 9, 2020 at
> > 10:21 AM
> > >> >> Maximilian Michels
> > >> >> > > <mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > >> >> > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > >> >> > > >> > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > >> >> > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > >> <mailto:mxm@apache.org <ma...@apache.org>>>
> > >> >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > >> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > >> >> > > >> >
> > >> >> > > >> > Hi,
> > >> >> > > >> >
> > >> >> > > >> > We recently saw an
> > >> increase in
> > >> >> latency
> > >> >> > migrating from Beam
> > >> >> > > 2.18.0 to
> > >> >> > > >> > 2.21.0 (Python SDK
> > with Flink
> > >> >> Runner). This
> > >> >> > proofed very
> > >> >> > > hard to debug
> > >> >> > > >> > and it looks like
> each
> > >> version in
> > >> >> between
> > >> >> > the two versions
> > >> >> > > let to
> > >> >> > > >> > increased latency.
> > >> >> > > >> >
> > >> >> > > >> > This is not the
> > first time
> > >> we saw
> > >> >> issues
> > >> >> > when migrating,
> > >> >> > > another
> > >> >> > > >> > time we
> > >> >> > > >> > had a decline in
> > >> checkpointing
> > >> >> performance
> > >> >> > and thus added a
> > >> >> > > >> > checkpointing test
> > [1] and
> > >> dashboard
> > >> >> [2] (see
> > >> >> > > checkpointing widget).
> > >> >> > > >> >
> > >> >> > > >> > That makes me wonder
> > if we
> > >> should
> > >> >> monitor
> > >> >> > performance
> > >> >> > > (throughput /
> > >> >> > > >> > latency) for basic
> > use cases
> > >> as part
> > >> >> of the
> > >> >> > release
> > >> >> > > testing. Currently,
> > >> >> > > >> > our release guide [3]
> > >> mentions
> > >> >> running
> > >> >> > examples but not
> > >> >> > > evaluating the
> > >> >> > > >> > performance. I think
> it
> > >> would be good
> > >> >> > practice to check
> > >> >> > > relevant charts
> > >> >> > > >> > with performance
> > >> measurements as
> > >> >> part of of
> > >> >> > the release
> > >> >> > > process. The
> > >> >> > > >> > release guide should
> > reflect
> > >> that.
> > >> >> > > >> >
> > >> >> > > >> > WDYT?
> > >> >> > > >> >
> > >> >> > > >> > -Max
> > >> >> > > >> >
> > >> >> > > >> > PS: Of course, this
> > requires
> > >> tests
> > >> >> and
> > >> >> > metrics to be
> > >> >> > > available. This PR
> > >> >> > > >> > adds latency
> > measurements
> > >> to the
> > >> >> load tests
> > >> >> > [4].
> > >> >> > > >> >
> > >> >> > > >> >
> > >> >> > > >> > [1]
> > >> >> https://github.com/apache/beam/pull/11558
> > >> >> > > >> > [2]
> > >> >> > > >> >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >>
> > >>
> > >>
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
> >
> > >>
> > >>
> > >> >>
> > >> >> > > >> > [3]
> > >> >> > https://beam.apache.org/contribute/release-guide/
> > >> >> > > >> > [4]
> > >> >> https://github.com/apache/beam/pull/12065
> > >> >> > > >> >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >>
> > >> >> Michał Walenia
> > >> >> Polidea <https://www.polidea.com/> | Software Engineer
> > >> >>
> > >> >> M: +48 791 432 002 <tel:+48%20791%20432%20002>
> > <tel:+48%20791%20432%20002> <tel:+48791432002
> > <tel:+48%20791%20432%20002>
> > >> <tel:+48%20791%20432%20002>>
> > >> >> E: michal.walenia@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:michal.walenia@polidea.com
> > <ma...@polidea.com>>
> > >> <mailto:michal.walenia@polidea.com
> > <ma...@polidea.com>
> > >> <mailto:michal.walenia@polidea.com
> > <ma...@polidea.com>>>
> > >> >>
> > >> >> Unique Tech
> > >> >> Check out our projects! <
> https://www.polidea.com/our-work>
> > >> >>
> > >>
> >
>
--
Tobiasz Kędzierski
Polidea <https://www.polidea.com/> | Junior Software Engineer
E: tobiasz.kedzierski@polidea.com
[image: Polidea] <https://www.polidea.com/>
Check out our projects! <https://www.polidea.com/our-work>
[image: Github] <https://github.com/Polidea> [image: Facebook]
<https://www.facebook.com/Polidea.Software> [image: Twitter]
<https://twitter.com/polidea> [image: Linkedin]
<https://www.linkedin.com/company/polidea> [image: Instagram]
<https://instagram.com/polidea> [image: Behance]
<https://www.behance.net/polidea> [image: dribbble]
<https://dribbble.com/polideadesign>
Re: Monitoring performance for releases
Posted by Maximilian Michels <mx...@apache.org>.
Robert, this is not too far off what I'm proposing. We can always create
JIRA issues for performance regressions and mark them with a Fix
Version. Especially, the time of the release is a good time to
re-evaluate whether some gross performance regressions can be detected.
Of course, if it's a more gradual and less noticeable once, we might
miss it.
I agree that good performance is a continuous effort, but many times
there are actual problems which can be dealt with at the time of the
release. Those might be very hard to fix a couple of releases down the
line because they are layered by new problems and much harder to find.
-Max
On 03.08.20 22:17, Robert Bradshaw wrote:
> I have to admit I still have some qualms about tying detecting and
> fixing performance regressions as part of the release process (which is
> onerous enough as it is). Instead, I think we'd be better off with a
> separate process to detect and triage performance issues, which, when
> they occur, may merit filing a blocker which will require fixing before
> the release just like any other blocker would. Hopefully this would
> result in issues being detected (and resolved) sooner.
>
> That being said, if a release is known to have performance regressions,
> that should be called out when the RCs are cut, and if not resolved,
> probably as part of the release notes as well.
>
> On Mon, Aug 3, 2020 at 9:40 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Here a first version of the updated release guide:
> https://github.com/apache/beam/pull/12455
>
> Feel free to comment.
>
> -Max
>
> On 29.07.20 17:27, Maximilian Michels wrote:
> > Thanks! I'm following up with this PR to display the Flink Pardo
> > streaming data: https://github.com/apache/beam/pull/12408
> >
> > Streaming data appears to be missing for Dataflow. We can revise the
> > Jenkins jobs to add those.
> >
> > -Max
> >
> > On 29.07.20 17:01, Tyson Hamilton wrote:
> >> Max,
> >>
> >> The runner dimension are present when hovering over a particular
> >> graph. For some more info, the load test configurations can be
> found
> >> here [1]. I didn't get a chance to look into them but there are
> tests
> >> for all the runners there, possibly not for every loadtest.
> >>
> >> [1]: https://github.com/apache/beam/tree/master/.test-infra/jenkins
> >>
> >> -Tyson
> >>
> >> On Wed, Jul 29, 2020 at 3:46 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>
> >> Looks like the permissions won't be necessary because backup
> data
> >> gets
> >> loaded into the local InfluxDb instance which makes writing
> queries
> >> locally possible.
> >>
> >> On 29.07.20 12:21, Maximilian Michels wrote:
> >> > Thanks Michał!
> >> >
> >> > It is a bit tricky to verify the exported query works if
> I don't
> >> have
> >> > access to the data stored in InfluxDb.
> >> >
> >> > ==> Could somebody give me permissions to
> max.michels@gmail.com <ma...@gmail.com>
> >> <mailto:max.michels@gmail.com
> <ma...@gmail.com>> for
> >> > apache-beam-testing such that I can setup a ssh
> port-forwarding
> >> from the
> >> > InfluxDb pod to my machine? I do have access to see the
> pods but
> >> that is
> >> > not enough.
> >> >
> >> >> I think that the only test data is from Python streaming
> tests,
> >> which
> >> >> are not implemented right now (check out
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
>
> >>
> >>
> >> >> )
> >> >
> >> > Additionally, there is an entire dimension missing:
> Runners. I'm
> >> > assuming this data is for Dataflow?
> >> >
> >> > -Max
> >> >
> >> > On 29.07.20 11:55, Michał Walenia wrote:
> >> >> Hi there,
> >> >>
> >> >> > Indeed the Python load test data appears to be missing:
> >> >> >
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
>
> >>
> >>
> >> >>
> >> >>
> >> >> I think that the only test data is from Python streaming
> tests,
> >> which
> >> >> are not implemented right now (check out
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
>
> >>
> >>
> >> >> )
> >> >>
> >> >> As for updating the dashboards, the manual for doing
> this is
> >> here:
> >> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-UpdatingDashboards
>
> >>
> >>
> >> >>
> >> >>
> >> >> I hope this helps,
> >> >>
> >> >> Michal
> >> >>
> >> >> On Mon, Jul 27, 2020 at 4:31 PM Maximilian Michels
> >> <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> >> >>
> >> >> Indeed the Python load test data appears to be missing:
> >> >>
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
>
> >>
> >>
> >> >>
> >> >>
> >> >> How do we typically modify the dashboards?
> >> >>
> >> >> It looks like we need to edit this json file:
> >> >>
> >> >>
> >>
> >>
> https://github.com/apache/beam/blob/8d460db620d2ff1257b0e092218294df15b409a1/.test-infra/metrics/grafana/dashboards/perftests_metrics/ParDo_Load_Tests.json#L81
>
> >>
> >>
> >> >>
> >> >>
> >> >> I found some documentation on the deployment:
> >> >>
> >> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/BEAM/Test+Results+Monitoring
> >> >>
> >> >>
> >> >> +1 for alerting or weekly emails including performance
> >> numbers for
> >> >> fixed
> >> >> intervals (1d, 1w, 1m, previous release).
> >> >>
> >> >> +1 for linking the dashboards in the release guide
> to allow
> >> for a
> >> >> comparison as part of the release process.
> >> >>
> >> >> As a first step, consolidating all the data seems
> like the
> >> most
> >> >> pressing
> >> >> problem to solve.
> >> >>
> >> >> @Kamil I could need some advice regarding how to proceed
> >> updating the
> >> >> dashboards.
> >> >>
> >> >> -Max
> >> >>
> >> >> On 22.07.20 20:20, Robert Bradshaw wrote:
> >> >> > On Tue, Jul 21, 2020 at 9:58 AM Thomas Weise
> >> <thw@apache.org <ma...@apache.org>
> <mailto:thw@apache.org <ma...@apache.org>>
> >> >> <mailto:thw@apache.org <ma...@apache.org>
> <mailto:thw@apache.org <ma...@apache.org>>>
> >> >> > <mailto:thw@apache.org <ma...@apache.org>
> <mailto:thw@apache.org <ma...@apache.org>>
> >> <mailto:thw@apache.org <ma...@apache.org>
> <mailto:thw@apache.org <ma...@apache.org>>>>> wrote:
> >> >> >
> >> >> > It appears that there is coverage missing in the
> >> Grafana
> >> >> dashboards
> >> >> > (it could also be that I just don't find it).
> >> >> >
> >> >> > For example:
> >> >> >
> >> >>
> >> >>
> >>
> >>
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
>
> >>
> >>
> >> >>
> >> >> >
> >> >> > The GBK and ParDo tests have a selection for
> {batch,
> >> >> streaming} and
> >> >> > SDK. No coverage for streaming and python?
> There is
> >> also no
> >> >> runner
> >> >> > option currently.
> >> >> >
> >> >> > We have seen repeated regressions with
> streaming,
> >> Python,
> >> >> Flink. The
> >> >> > test has been contributed. It would be great
> if the
> >> results
> >> >> can be
> >> >> > covered as part of release verification.
> >> >> >
> >> >> >
> >> >> > Even better would be if we can use these
> dashboards (plus
> >> >> alerting or
> >> >> > similar?) to find issues before release verification.
> >> It's much
> >> >> easier
> >> >> > to fix things earlier.
> >> >> >
> >> >> >
> >> >> > Thomas
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Jul 21, 2020 at 7:55 AM Kamil Wasilewski
> >> >> > <kamil.wasilewski@polidea.com
> <ma...@polidea.com>
> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>>
> >> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>
> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>>>
> >> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>
> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>>
> >> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>
> >> <mailto:kamil.wasilewski@polidea.com
> <ma...@polidea.com>>>>>
> >> >> > wrote:
> >> >> >
> >> >> > The prerequisite is that we have all the
> >> stats in one
> >> >> place.
> >> >> > They seem
> >> >> > to be scattered across
> >> >> http://metrics.beam.apache.org and
> >> >> > https://apache-beam-testing.appspot.com.
> >> >> >
> >> >> > Would it be possible to consolidate
> the two,
> >> i.e.
> >> >> use the
> >> >> > Grafana-based
> >> >> > dashboard to load the legacy stats?
> >> >> >
> >> >> >
> >> >> > I'm pretty sure that all dashboards have been
> >> moved to
> >> >> > http://metrics.beam.apache.org. Let me know if I
> missed
> >> >> > something during the migration.
> >> >> >
> >> >> > I think we should turn off
> >> >> > https://apache-beam-testing.appspot.com in the near
> >> future. New
> >> >> > Grafana-based dashboards have been working
> >> seamlessly for
> >> >> some
> >> >> > time now and there's no point in
> maintaining the
> >> older
> >> >> solution.
> >> >> > We'd also avoid ambiguity in where the stats
> >> should be
> >> >> looked for.
> >> >> >
> >> >> > Kamil
> >> >> >
> >> >> > On Tue, Jul 21, 2020 at 4:17 PM
> Maximilian Michels
> >> >> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> >> >> >
> >> >> > > It doesn't support https. I had to
> add an
> >> >> exception to
> >> >> > the HTTPS Everywhere extension for
> >> >> "metrics.beam.apache.org
> <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>".
> >> >> >
> >> >> > *facepalm* Thanks Udi! It would
> always hang
> >> on me
> >> >> because I
> >> >> > use HTTPS
> >> >> > Everywhere.
> >> >> >
> >> >> > > To be explicit, I am supporting
> the idea of
> >> >> reviewing the
> >> >> > release guide but not changing the
> release
> >> process
> >> >> for the
> >> >> > already in-progress release.
> >> >> >
> >> >> > I consider the release guide
> immutable for the
> >> >> process of a
> >> >> > release.
> >> >> > Thus, a change to the release guide
> can only
> >> affect
> >> >> new
> >> >> > upcoming
> >> >> > releases, not an in-process release.
> >> >> >
> >> >> > > +1 and I think we can also evaluate
> >> whether flaky
> >> >> tests
> >> >> > should be reviewed as release blockers or
> >> not. Some
> >> >> flaky
> >> >> > tests would be hiding real issues our
> users
> >> could
> >> >> face.
> >> >> >
> >> >> > Flaky tests are also worth to take into
> >> account when
> >> >> > releasing, but a
> >> >> > little harder to find because may
> just happen
> >> to pass
> >> >> during
> >> >> > building
> >> >> > the release. It is possible though if we
> >> strictly
> >> >> capture
> >> >> > flaky tests
> >> >> > via JIRA and mark them with the Fix
> Version
> >> for the
> >> >> release.
> >> >> >
> >> >> > > We keep accumulating dashboards and
> >> >> > > tests that few people care about,
> so it is
> >> >> probably worth
> >> >> > that we use
> >> >> > > them or get a way to alert us of
> >> regressions
> >> >> during the
> >> >> > release cycle
> >> >> > > to catch this even before the RCs.
> >> >> >
> >> >> > +1 The release guide should be
> explicit about
> >> which
> >> >> > performance test
> >> >> > results to evaluate.
> >> >> >
> >> >> > The prerequisite is that we have all the
> >> stats in one
> >> >> place.
> >> >> > They seem
> >> >> > to be scattered across
> >> >> http://metrics.beam.apache.org and
> >> >> > https://apache-beam-testing.appspot.com.
> >> >> >
> >> >> > Would it be possible to consolidate
> the two,
> >> i.e.
> >> >> use the
> >> >> > Grafana-based
> >> >> > dashboard to load the legacy stats?
> >> >> >
> >> >> > For the evaluation during the release
> >> process, I
> >> >> suggest to
> >> >> > use a
> >> >> > standardized set of performance tests
> for all
> >> >> runners, e.g.:
> >> >> >
> >> >> > - Nexmark
> >> >> > - ParDo (Classic/Portable)
> >> >> > - GroupByKey
> >> >> > - IO
> >> >> >
> >> >> >
> >> >> > -Max
> >> >> >
> >> >> > On 21.07.20 01:23, Ahmet Altay wrote:
> >> >> > >
> >> >> > > On Mon, Jul 20, 2020 at 3:07 PM
> Ismaël
> >> Mejía
> >> >> > <iemejia@gmail.com
> <ma...@gmail.com> <mailto:iemejia@gmail.com
> <ma...@gmail.com>>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>>
> >> >> > > <mailto:iemejia@gmail.com
> <ma...@gmail.com>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>>>>
> >> wrote:
> >> >> > >
> >> >> > > +1
> >> >> > >
> >> >> > > This is not in the release
> guide and
> >> we should
> >> >> > probably re evaluate if
> >> >> > > this should be a release blocking
> >> reason.
> >> >> > > Of course exceptionally a
> performance
> >> >> regression
> >> >> > could be motivated by
> >> >> > > a correctness fix or a worth
> refactor,
> >> so we
> >> >> should
> >> >> > consider this.
> >> >> > >
> >> >> > >
> >> >> > > +1 and I think we can also
> evaluate whether
> >> >> flaky tests
> >> >> > should be
> >> >> > > reviewed as release blockers or
> not. Some
> >> flaky
> >> >> tests
> >> >> > would be hiding
> >> >> > > real issues our users could face.
> >> >> > >
> >> >> > > To be explicit, I am supporting
> the idea of
> >> >> reviewing the
> >> >> > release guide
> >> >> > > but not changing the release
> process for
> >> the
> >> >> already
> >> >> > in-progress release.
> >> >> > >
> >> >> > >
> >> >> > > We have been tracking and fixing
> >> performance
> >> >> > regressions multiple
> >> >> > > times found simply by checking the
> >> nexmark
> >> >> tests
> >> >> > including on the
> >> >> > > ongoing 2.23.0 release so
> value is
> >> there.
> >> >> Nexmark
> >> >> > does not cover yet
> >> >> > > python and portable runners so
> we are
> >> probably
> >> >> still
> >> >> > missing many
> >> >> > > issues and it is worth to work on
> >> this. In any
> >> >> case
> >> >> > we should probably
> >> >> > > decide what validations
> matter. We keep
> >> >> accumulating
> >> >> > dashboards and
> >> >> > > tests that few people care
> about, so
> >> it is
> >> >> probably
> >> >> > worth that we use
> >> >> > > them or get a way to alert us of
> >> regressions
> >> >> during
> >> >> > the release cycle
> >> >> > > to catch this even before the RCs.
> >> >> > >
> >> >> > >
> >> >> > > I agree. And if we cannot use
> >> dashboards/tests in a
> >> >> > meaningful way, IMO
> >> >> > > we can remove them. There is not much
> >> value to
> >> >> maintain
> >> >> > them if they do
> >> >> > > not provide important signals.
> >> >> > >
> >> >> > >
> >> >> > > On Fri, Jul 10, 2020 at 9:30
> PM Udi
> >> Meiri
> >> >> > <ehudm@google.com
> <ma...@google.com> <mailto:ehudm@google.com
> <ma...@google.com>>
> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>>
> >> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>
> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>>>
> >> >> > > <mailto:ehudm@google.com
> <ma...@google.com>
> >> <mailto:ehudm@google.com <ma...@google.com>>
> >> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>>
> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>
> >> >> <mailto:ehudm@google.com <ma...@google.com>
> <mailto:ehudm@google.com <ma...@google.com>>>>>>
> >> >> > wrote:
> >> >> > > >
> >> >> > > > On Thu, Jul 9, 2020 at 12:48 PM
> >> Maximilian
> >> >> Michels
> >> >> > > <mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> >> >> > <mailto:mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> >> >> > > >>
> >> >> > > >> Not yet, I just learned
> about the
> >> >> migration to a
> >> >> > new frontend,
> >> >> > > including
> >> >> > > >> a new backend (InfluxDB
> instead of
> >> >> BigQuery).
> >> >> > > >>
> >> >> > > >> > - Are the metrics
> available on
> >> >> > metrics.beam.apache.org
> <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>?
> >> >> > > >>
> >> >> > > >> Is
> http://metrics.beam.apache.org
> >> online?
> >> >> I was
> >> >> > never able to
> >> >> > > access it.
> >> >> > > >
> >> >> > > >
> >> >> > > > It doesn't support https. I
> had to
> >> add an
> >> >> > exception to the HTTPS
> >> >> > > Everywhere extension for
> >> >> "metrics.beam.apache.org
> <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>".
> >> >> > > >
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> > - What is the feature delta
> >> between
> >> >> usinig
> >> >> > > metrics.beam.apache.org
> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org> (much
> >> >> > > better UI) and using
> >> >> apache-beam-testing.appspot.com
> <http://apache-beam-testing.appspot.com>
> >> <http://apache-beam-testing.appspot.com>
> >> >> <http://apache-beam-testing.appspot.com>
> >> >> > <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>?
> >> >> > > >>
> >> >> > > >> AFAIK it is an ongoing
> migration
> >> and the
> >> >> delta
> >> >> > appears to be high.
> >> >> > > >>
> >> >> > > >> > - Can we notice regressions
> >> faster than
> >> >> > release cadence?
> >> >> > > >>
> >> >> > > >> Absolutely! A report with the
> >> latest
> >> >> numbers
> >> >> > including
> >> >> > > statistics about
> >> >> > > >> the growth of metrics
> would be
> >> useful.
> >> >> > > >>
> >> >> > > >> > - Can we get automated
> alerts?
> >> >> > > >>
> >> >> > > >> I think we could setup a
> Jenkins
> >> job to do
> >> >> this.
> >> >> > > >>
> >> >> > > >> -Max
> >> >> > > >>
> >> >> > > >> On 09.07.20 20:26, Kenneth
> Knowles
> >> wrote:
> >> >> > > >> > Questions:
> >> >> > > >> >
> >> >> > > >> > - Are the metrics
> available on
> >> >> > metrics.beam.apache.org
> <http://metrics.beam.apache.org> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>
> >> >> > > >> >
> <http://metrics.beam.apache.org>?
> >> >> > > >> > - What is the feature
> delta
> >> between
> >> >> usinig
> >> >> > > metrics.beam.apache.org
> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>
> >> >> > > >> >
> <http://metrics.beam.apache.org>
> >> (much
> >> >> better
> >> >> > UI) and using
> >> >> > > >> >
> apache-beam-testing.appspot.com <http://apache-beam-testing.appspot.com>
> >> <http://apache-beam-testing.appspot.com>
> >> >> <http://apache-beam-testing.appspot.com>
> >> >> > <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>?
> >> >> > > >> > - Can we notice
> regressions
> >> faster than
> >> >> > release cadence?
> >> >> > > >> > - Can we get automated
> alerts?
> >> >> > > >> >
> >> >> > > >> > Kenn
> >> >> > > >> >
> >> >> > > >> > On Thu, Jul 9, 2020 at
> 10:21 AM
> >> >> Maximilian Michels
> >> >> > > <mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> >> >> > <mailto:mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> >> >> > > >> > <mailto:mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> >> >> > <mailto:mxm@apache.org
> <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> >> >> > > >> >
> >> >> > > >> > Hi,
> >> >> > > >> >
> >> >> > > >> > We recently saw an
> >> increase in
> >> >> latency
> >> >> > migrating from Beam
> >> >> > > 2.18.0 to
> >> >> > > >> > 2.21.0 (Python SDK
> with Flink
> >> >> Runner). This
> >> >> > proofed very
> >> >> > > hard to debug
> >> >> > > >> > and it looks like each
> >> version in
> >> >> between
> >> >> > the two versions
> >> >> > > let to
> >> >> > > >> > increased latency.
> >> >> > > >> >
> >> >> > > >> > This is not the
> first time
> >> we saw
> >> >> issues
> >> >> > when migrating,
> >> >> > > another
> >> >> > > >> > time we
> >> >> > > >> > had a decline in
> >> checkpointing
> >> >> performance
> >> >> > and thus added a
> >> >> > > >> > checkpointing test
> [1] and
> >> dashboard
> >> >> [2] (see
> >> >> > > checkpointing widget).
> >> >> > > >> >
> >> >> > > >> > That makes me wonder
> if we
> >> should
> >> >> monitor
> >> >> > performance
> >> >> > > (throughput /
> >> >> > > >> > latency) for basic
> use cases
> >> as part
> >> >> of the
> >> >> > release
> >> >> > > testing. Currently,
> >> >> > > >> > our release guide [3]
> >> mentions
> >> >> running
> >> >> > examples but not
> >> >> > > evaluating the
> >> >> > > >> > performance. I think it
> >> would be good
> >> >> > practice to check
> >> >> > > relevant charts
> >> >> > > >> > with performance
> >> measurements as
> >> >> part of of
> >> >> > the release
> >> >> > > process. The
> >> >> > > >> > release guide should
> reflect
> >> that.
> >> >> > > >> >
> >> >> > > >> > WDYT?
> >> >> > > >> >
> >> >> > > >> > -Max
> >> >> > > >> >
> >> >> > > >> > PS: Of course, this
> requires
> >> tests
> >> >> and
> >> >> > metrics to be
> >> >> > > available. This PR
> >> >> > > >> > adds latency
> measurements
> >> to the
> >> >> load tests
> >> >> > [4].
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >> > [1]
> >> >> https://github.com/apache/beam/pull/11558
> >> >> > > >> > [2]
> >> >> > > >> >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >>
> >>
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
>
> >>
> >>
> >> >>
> >> >> > > >> > [3]
> >> >> > https://beam.apache.org/contribute/release-guide/
> >> >> > > >> > [4]
> >> >> https://github.com/apache/beam/pull/12065
> >> >> > > >> >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Michał Walenia
> >> >> Polidea <https://www.polidea.com/> | Software Engineer
> >> >>
> >> >> M: +48 791 432 002 <tel:+48%20791%20432%20002>
> <tel:+48%20791%20432%20002> <tel:+48791432002
> <tel:+48%20791%20432%20002>
> >> <tel:+48%20791%20432%20002>>
> >> >> E: michal.walenia@polidea.com
> <ma...@polidea.com>
> >> <mailto:michal.walenia@polidea.com
> <ma...@polidea.com>>
> >> <mailto:michal.walenia@polidea.com
> <ma...@polidea.com>
> >> <mailto:michal.walenia@polidea.com
> <ma...@polidea.com>>>
> >> >>
> >> >> Unique Tech
> >> >> Check out our projects! <https://www.polidea.com/our-work>
> >> >>
> >>
>
Re: Monitoring performance for releases
Posted by Robert Bradshaw <ro...@google.com>.
I have to admit I still have some qualms about tying detecting and fixing
performance regressions as part of the release process (which is onerous
enough as it is). Instead, I think we'd be better off with a
separate process to detect and triage performance issues, which, when they
occur, may merit filing a blocker which will require fixing before the
release just like any other blocker would. Hopefully this would result in
issues being detected (and resolved) sooner.
That being said, if a release is known to have performance regressions,
that should be called out when the RCs are cut, and if not resolved,
probably as part of the release notes as well.
On Mon, Aug 3, 2020 at 9:40 AM Maximilian Michels <mx...@apache.org> wrote:
> Here a first version of the updated release guide:
> https://github.com/apache/beam/pull/12455
>
> Feel free to comment.
>
> -Max
>
> On 29.07.20 17:27, Maximilian Michels wrote:
> > Thanks! I'm following up with this PR to display the Flink Pardo
> > streaming data: https://github.com/apache/beam/pull/12408
> >
> > Streaming data appears to be missing for Dataflow. We can revise the
> > Jenkins jobs to add those.
> >
> > -Max
> >
> > On 29.07.20 17:01, Tyson Hamilton wrote:
> >> Max,
> >>
> >> The runner dimension are present when hovering over a particular
> >> graph. For some more info, the load test configurations can be found
> >> here [1]. I didn't get a chance to look into them but there are tests
> >> for all the runners there, possibly not for every loadtest.
> >>
> >> [1]: https://github.com/apache/beam/tree/master/.test-infra/jenkins
> >>
> >> -Tyson
> >>
> >> On Wed, Jul 29, 2020 at 3:46 AM Maximilian Michels <mxm@apache.org
> >> <ma...@apache.org>> wrote:
> >>
> >> Looks like the permissions won't be necessary because backup data
> >> gets
> >> loaded into the local InfluxDb instance which makes writing queries
> >> locally possible.
> >>
> >> On 29.07.20 12:21, Maximilian Michels wrote:
> >> > Thanks Michał!
> >> >
> >> > It is a bit tricky to verify the exported query works if I don't
> >> have
> >> > access to the data stored in InfluxDb.
> >> >
> >> > ==> Could somebody give me permissions to max.michels@gmail.com
> >> <ma...@gmail.com> for
> >> > apache-beam-testing such that I can setup a ssh port-forwarding
> >> from the
> >> > InfluxDb pod to my machine? I do have access to see the pods but
> >> that is
> >> > not enough.
> >> >
> >> >> I think that the only test data is from Python streaming tests,
> >> which
> >> >> are not implemented right now (check out
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
> >>
> >>
> >> >> )
> >> >
> >> > Additionally, there is an entire dimension missing: Runners. I'm
> >> > assuming this data is for Dataflow?
> >> >
> >> > -Max
> >> >
> >> > On 29.07.20 11:55, Michał Walenia wrote:
> >> >> Hi there,
> >> >>
> >> >> > Indeed the Python load test data appears to be missing:
> >> >> >
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
> >>
> >>
> >> >>
> >> >>
> >> >> I think that the only test data is from Python streaming tests,
> >> which
> >> >> are not implemented right now (check out
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=batch&var-sdk=python
> >>
> >>
> >> >> )
> >> >>
> >> >> As for updating the dashboards, the manual for doing this is
> >> here:
> >> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-UpdatingDashboards
> >>
> >>
> >> >>
> >> >>
> >> >> I hope this helps,
> >> >>
> >> >> Michal
> >> >>
> >> >> On Mon, Jul 27, 2020 at 4:31 PM Maximilian Michels
> >> <mxm@apache.org <ma...@apache.org>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >> >>
> >> >> Indeed the Python load test data appears to be missing:
> >> >>
> >> >>
> >>
> >>
> http://metrics.beam.apache.org/d/MOi-kf3Zk/pardo-load-tests?orgId=1&var-processingType=streaming&var-sdk=python
> >>
> >>
> >> >>
> >> >>
> >> >> How do we typically modify the dashboards?
> >> >>
> >> >> It looks like we need to edit this json file:
> >> >>
> >> >>
> >>
> >>
> https://github.com/apache/beam/blob/8d460db620d2ff1257b0e092218294df15b409a1/.test-infra/metrics/grafana/dashboards/perftests_metrics/ParDo_Load_Tests.json#L81
> >>
> >>
> >> >>
> >> >>
> >> >> I found some documentation on the deployment:
> >> >>
> >> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/BEAM/Test+Results+Monitoring
> >> >>
> >> >>
> >> >> +1 for alerting or weekly emails including performance
> >> numbers for
> >> >> fixed
> >> >> intervals (1d, 1w, 1m, previous release).
> >> >>
> >> >> +1 for linking the dashboards in the release guide to allow
> >> for a
> >> >> comparison as part of the release process.
> >> >>
> >> >> As a first step, consolidating all the data seems like the
> >> most
> >> >> pressing
> >> >> problem to solve.
> >> >>
> >> >> @Kamil I could need some advice regarding how to proceed
> >> updating the
> >> >> dashboards.
> >> >>
> >> >> -Max
> >> >>
> >> >> On 22.07.20 20:20, Robert Bradshaw wrote:
> >> >> > On Tue, Jul 21, 2020 at 9:58 AM Thomas Weise
> >> <thw@apache.org <ma...@apache.org>
> >> >> <mailto:thw@apache.org <ma...@apache.org>>
> >> >> > <mailto:thw@apache.org <ma...@apache.org>
> >> <mailto:thw@apache.org <ma...@apache.org>>>> wrote:
> >> >> >
> >> >> > It appears that there is coverage missing in the
> >> Grafana
> >> >> dashboards
> >> >> > (it could also be that I just don't find it).
> >> >> >
> >> >> > For example:
> >> >> >
> >> >>
> >> >>
> >>
> >>
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
> >>
> >>
> >> >>
> >> >> >
> >> >> > The GBK and ParDo tests have a selection for {batch,
> >> >> streaming} and
> >> >> > SDK. No coverage for streaming and python? There is
> >> also no
> >> >> runner
> >> >> > option currently.
> >> >> >
> >> >> > We have seen repeated regressions with streaming,
> >> Python,
> >> >> Flink. The
> >> >> > test has been contributed. It would be great if the
> >> results
> >> >> can be
> >> >> > covered as part of release verification.
> >> >> >
> >> >> >
> >> >> > Even better would be if we can use these dashboards (plus
> >> >> alerting or
> >> >> > similar?) to find issues before release verification.
> >> It's much
> >> >> easier
> >> >> > to fix things earlier.
> >> >> >
> >> >> >
> >> >> > Thomas
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Jul 21, 2020 at 7:55 AM Kamil Wasilewski
> >> >> > <kamil.wasilewski@polidea.com
> >> <ma...@polidea.com>
> >> >> <mailto:kamil.wasilewski@polidea.com
> >> <ma...@polidea.com>>
> >> >> <mailto:kamil.wasilewski@polidea.com
> >> <ma...@polidea.com>
> >> >> <mailto:kamil.wasilewski@polidea.com
> >> <ma...@polidea.com>>>>
> >> >> > wrote:
> >> >> >
> >> >> > The prerequisite is that we have all the
> >> stats in one
> >> >> place.
> >> >> > They seem
> >> >> > to be scattered across
> >> >> http://metrics.beam.apache.org and
> >> >> > https://apache-beam-testing.appspot.com.
> >> >> >
> >> >> > Would it be possible to consolidate the two,
> >> i.e.
> >> >> use the
> >> >> > Grafana-based
> >> >> > dashboard to load the legacy stats?
> >> >> >
> >> >> >
> >> >> > I'm pretty sure that all dashboards have been
> >> moved to
> >> >> > http://metrics.beam.apache.org. Let me know if I missed
> >> >> > something during the migration.
> >> >> >
> >> >> > I think we should turn off
> >> >> > https://apache-beam-testing.appspot.com in the near
> >> future. New
> >> >> > Grafana-based dashboards have been working
> >> seamlessly for
> >> >> some
> >> >> > time now and there's no point in maintaining the
> >> older
> >> >> solution.
> >> >> > We'd also avoid ambiguity in where the stats
> >> should be
> >> >> looked for.
> >> >> >
> >> >> > Kamil
> >> >> >
> >> >> > On Tue, Jul 21, 2020 at 4:17 PM Maximilian
> Michels
> >> >> > <mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> >> >> >
> >> >> > > It doesn't support https. I had to add an
> >> >> exception to
> >> >> > the HTTPS Everywhere extension for
> >> >> "metrics.beam.apache.org <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>".
> >> >> >
> >> >> > *facepalm* Thanks Udi! It would always hang
> >> on me
> >> >> because I
> >> >> > use HTTPS
> >> >> > Everywhere.
> >> >> >
> >> >> > > To be explicit, I am supporting the idea
> of
> >> >> reviewing the
> >> >> > release guide but not changing the release
> >> process
> >> >> for the
> >> >> > already in-progress release.
> >> >> >
> >> >> > I consider the release guide immutable for
> the
> >> >> process of a
> >> >> > release.
> >> >> > Thus, a change to the release guide can only
> >> affect
> >> >> new
> >> >> > upcoming
> >> >> > releases, not an in-process release.
> >> >> >
> >> >> > > +1 and I think we can also evaluate
> >> whether flaky
> >> >> tests
> >> >> > should be reviewed as release blockers or
> >> not. Some
> >> >> flaky
> >> >> > tests would be hiding real issues our users
> >> could
> >> >> face.
> >> >> >
> >> >> > Flaky tests are also worth to take into
> >> account when
> >> >> > releasing, but a
> >> >> > little harder to find because may just happen
> >> to pass
> >> >> during
> >> >> > building
> >> >> > the release. It is possible though if we
> >> strictly
> >> >> capture
> >> >> > flaky tests
> >> >> > via JIRA and mark them with the Fix Version
> >> for the
> >> >> release.
> >> >> >
> >> >> > > We keep accumulating dashboards and
> >> >> > > tests that few people care about, so it is
> >> >> probably worth
> >> >> > that we use
> >> >> > > them or get a way to alert us of
> >> regressions
> >> >> during the
> >> >> > release cycle
> >> >> > > to catch this even before the RCs.
> >> >> >
> >> >> > +1 The release guide should be explicit about
> >> which
> >> >> > performance test
> >> >> > results to evaluate.
> >> >> >
> >> >> > The prerequisite is that we have all the
> >> stats in one
> >> >> place.
> >> >> > They seem
> >> >> > to be scattered across
> >> >> http://metrics.beam.apache.org and
> >> >> > https://apache-beam-testing.appspot.com.
> >> >> >
> >> >> > Would it be possible to consolidate the two,
> >> i.e.
> >> >> use the
> >> >> > Grafana-based
> >> >> > dashboard to load the legacy stats?
> >> >> >
> >> >> > For the evaluation during the release
> >> process, I
> >> >> suggest to
> >> >> > use a
> >> >> > standardized set of performance tests for all
> >> >> runners, e.g.:
> >> >> >
> >> >> > - Nexmark
> >> >> > - ParDo (Classic/Portable)
> >> >> > - GroupByKey
> >> >> > - IO
> >> >> >
> >> >> >
> >> >> > -Max
> >> >> >
> >> >> > On 21.07.20 01:23, Ahmet Altay wrote:
> >> >> > >
> >> >> > > On Mon, Jul 20, 2020 at 3:07 PM Ismaël
> >> Mejía
> >> >> > <iemejia@gmail.com <mailto:iemejia@gmail.com
> >
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> >> >> > > <mailto:iemejia@gmail.com
> >> <ma...@gmail.com>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> >> <mailto:iemejia@gmail.com <ma...@gmail.com>
> >> >> <mailto:iemejia@gmail.com <ma...@gmail.com>>>>>
> >> wrote:
> >> >> > >
> >> >> > > +1
> >> >> > >
> >> >> > > This is not in the release guide and
> >> we should
> >> >> > probably re evaluate if
> >> >> > > this should be a release blocking
> >> reason.
> >> >> > > Of course exceptionally a performance
> >> >> regression
> >> >> > could be motivated by
> >> >> > > a correctness fix or a worth refactor,
> >> so we
> >> >> should
> >> >> > consider this.
> >> >> > >
> >> >> > >
> >> >> > > +1 and I think we can also evaluate
> whether
> >> >> flaky tests
> >> >> > should be
> >> >> > > reviewed as release blockers or not. Some
> >> flaky
> >> >> tests
> >> >> > would be hiding
> >> >> > > real issues our users could face.
> >> >> > >
> >> >> > > To be explicit, I am supporting the idea
> of
> >> >> reviewing the
> >> >> > release guide
> >> >> > > but not changing the release process for
> >> the
> >> >> already
> >> >> > in-progress release.
> >> >> > >
> >> >> > >
> >> >> > > We have been tracking and fixing
> >> performance
> >> >> > regressions multiple
> >> >> > > times found simply by checking the
> >> nexmark
> >> >> tests
> >> >> > including on the
> >> >> > > ongoing 2.23.0 release so value is
> >> there.
> >> >> Nexmark
> >> >> > does not cover yet
> >> >> > > python and portable runners so we are
> >> probably
> >> >> still
> >> >> > missing many
> >> >> > > issues and it is worth to work on
> >> this. In any
> >> >> case
> >> >> > we should probably
> >> >> > > decide what validations matter. We
> keep
> >> >> accumulating
> >> >> > dashboards and
> >> >> > > tests that few people care about, so
> >> it is
> >> >> probably
> >> >> > worth that we use
> >> >> > > them or get a way to alert us of
> >> regressions
> >> >> during
> >> >> > the release cycle
> >> >> > > to catch this even before the RCs.
> >> >> > >
> >> >> > >
> >> >> > > I agree. And if we cannot use
> >> dashboards/tests in a
> >> >> > meaningful way, IMO
> >> >> > > we can remove them. There is not much
> >> value to
> >> >> maintain
> >> >> > them if they do
> >> >> > > not provide important signals.
> >> >> > >
> >> >> > >
> >> >> > > On Fri, Jul 10, 2020 at 9:30 PM Udi
> >> Meiri
> >> >> > <ehudm@google.com <ma...@google.com>
> >> <mailto:ehudm@google.com <ma...@google.com>>
> >> >> <mailto:ehudm@google.com <ma...@google.com>
> >> <mailto:ehudm@google.com <ma...@google.com>>>
> >> >> > > <mailto:ehudm@google.com
> >> <ma...@google.com>
> >> >> <mailto:ehudm@google.com <ma...@google.com>>
> >> <mailto:ehudm@google.com <ma...@google.com>
> >> >> <mailto:ehudm@google.com <ma...@google.com>>>>>
> >> >> > wrote:
> >> >> > > >
> >> >> > > > On Thu, Jul 9, 2020 at 12:48 PM
> >> Maximilian
> >> >> Michels
> >> >> > > <mxm@apache.org
> >> <ma...@apache.org> <mailto:mxm@apache.org
> >> <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> > <mailto:mxm@apache.org
> >> <ma...@apache.org> <mailto:mxm@apache.org
> >> <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> >> >> > > >>
> >> >> > > >> Not yet, I just learned about the
> >> >> migration to a
> >> >> > new frontend,
> >> >> > > including
> >> >> > > >> a new backend (InfluxDB instead of
> >> >> BigQuery).
> >> >> > > >>
> >> >> > > >> > - Are the metrics available on
> >> >> > metrics.beam.apache.org <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>?
> >> >> > > >>
> >> >> > > >> Is http://metrics.beam.apache.org
> >> online?
> >> >> I was
> >> >> > never able to
> >> >> > > access it.
> >> >> > > >
> >> >> > > >
> >> >> > > > It doesn't support https. I had to
> >> add an
> >> >> > exception to the HTTPS
> >> >> > > Everywhere extension for
> >> >> "metrics.beam.apache.org <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>".
> >> >> > > >
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> > - What is the feature delta
> >> between
> >> >> usinig
> >> >> > > metrics.beam.apache.org
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org> (much
> >> >> > > better UI) and using
> >> >> apache-beam-testing.appspot.com
> >> <http://apache-beam-testing.appspot.com>
> >> >> <http://apache-beam-testing.appspot.com>
> >> >> > <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>?
> >> >> > > >>
> >> >> > > >> AFAIK it is an ongoing migration
> >> and the
> >> >> delta
> >> >> > appears to be high.
> >> >> > > >>
> >> >> > > >> > - Can we notice regressions
> >> faster than
> >> >> > release cadence?
> >> >> > > >>
> >> >> > > >> Absolutely! A report with the
> >> latest
> >> >> numbers
> >> >> > including
> >> >> > > statistics about
> >> >> > > >> the growth of metrics would be
> >> useful.
> >> >> > > >>
> >> >> > > >> > - Can we get automated alerts?
> >> >> > > >>
> >> >> > > >> I think we could setup a Jenkins
> >> job to do
> >> >> this.
> >> >> > > >>
> >> >> > > >> -Max
> >> >> > > >>
> >> >> > > >> On 09.07.20 20:26, Kenneth Knowles
> >> wrote:
> >> >> > > >> > Questions:
> >> >> > > >> >
> >> >> > > >> > - Are the metrics available on
> >> >> > metrics.beam.apache.org <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> >> > > <http://metrics.beam.apache.org>
> >> >> > > >> > <http://metrics.beam.apache.org
> >?
> >> >> > > >> > - What is the feature delta
> >> between
> >> >> usinig
> >> >> > > metrics.beam.apache.org
> >> <http://metrics.beam.apache.org>
> >> >> <http://metrics.beam.apache.org>
> >> <http://metrics.beam.apache.org>
> >> >> > <http://metrics.beam.apache.org>
> >> >> > > >> > <http://metrics.beam.apache.org
> >
> >> (much
> >> >> better
> >> >> > UI) and using
> >> >> > > >> > apache-beam-testing.appspot.com
> >> <http://apache-beam-testing.appspot.com>
> >> >> <http://apache-beam-testing.appspot.com>
> >> >> > <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>
> >> >> > >
> >> <http://apache-beam-testing.appspot.com>?
> >> >> > > >> > - Can we notice regressions
> >> faster than
> >> >> > release cadence?
> >> >> > > >> > - Can we get automated alerts?
> >> >> > > >> >
> >> >> > > >> > Kenn
> >> >> > > >> >
> >> >> > > >> > On Thu, Jul 9, 2020 at 10:21 AM
> >> >> Maximilian Michels
> >> >> > > <mxm@apache.org
> >> <ma...@apache.org> <mailto:mxm@apache.org
> >> <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> > <mailto:mxm@apache.org
> >> <ma...@apache.org> <mailto:mxm@apache.org
> >> <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>>
> >> >> > > >> > <mailto:mxm@apache.org
> >> <ma...@apache.org>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>>
> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>>>
> >> >> > <mailto:mxm@apache.org
> >> <ma...@apache.org> <mailto:mxm@apache.org
> >> <ma...@apache.org>>
> >> >> <mailto:mxm@apache.org <ma...@apache.org>
> >> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> >> >> > > >> >
> >> >> > > >> > Hi,
> >> >> > > >> >
> >> >> > > >> > We recently saw an
> >> increase in
> >> >> latency
> >> >> > migrating from Beam
> >> >> > > 2.18.0 to
> >> >> > > >> > 2.21.0 (Python SDK with
> Flink
> >> >> Runner). This
> >> >> > proofed very
> >> >> > > hard to debug
> >> >> > > >> > and it looks like each
> >> version in
> >> >> between
> >> >> > the two versions
> >> >> > > let to
> >> >> > > >> > increased latency.
> >> >> > > >> >
> >> >> > > >> > This is not the first time
> >> we saw
> >> >> issues
> >> >> > when migrating,
> >> >> > > another
> >> >> > > >> > time we
> >> >> > > >> > had a decline in
> >> checkpointing
> >> >> performance
> >> >> > and thus added a
> >> >> > > >> > checkpointing test [1] and
> >> dashboard
> >> >> [2] (see
> >> >> > > checkpointing widget).
> >> >> > > >> >
> >> >> > > >> > That makes me wonder if we
> >> should
> >> >> monitor
> >> >> > performance
> >> >> > > (throughput /
> >> >> > > >> > latency) for basic use cases
> >> as part
> >> >> of the
> >> >> > release
> >> >> > > testing. Currently,
> >> >> > > >> > our release guide [3]
> >> mentions
> >> >> running
> >> >> > examples but not
> >> >> > > evaluating the
> >> >> > > >> > performance. I think it
> >> would be good
> >> >> > practice to check
> >> >> > > relevant charts
> >> >> > > >> > with performance
> >> measurements as
> >> >> part of of
> >> >> > the release
> >> >> > > process. The
> >> >> > > >> > release guide should reflect
> >> that.
> >> >> > > >> >
> >> >> > > >> > WDYT?
> >> >> > > >> >
> >> >> > > >> > -Max
> >> >> > > >> >
> >> >> > > >> > PS: Of course, this requires
> >> tests
> >> >> and
> >> >> > metrics to be
> >> >> > > available. This PR
> >> >> > > >> > adds latency measurements
> >> to the
> >> >> load tests
> >> >> > [4].
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >> > [1]
> >> >> https://github.com/apache/beam/pull/11558
> >> >> > > >> > [2]
> >> >> > > >> >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >>
> >>
> https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
> >>
> >>
> >> >>
> >> >> > > >> > [3]
> >> >> > https://beam.apache.org/contribute/release-guide/
> >> >> > > >> > [4]
> >> >> https://github.com/apache/beam/pull/12065
> >> >> > > >> >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Michał Walenia
> >> >> Polidea <https://www.polidea.com/> | Software Engineer
> >> >>
> >> >> M: +48 791 432 002 <+48%20791%20432%20002>
> <tel:+48%20791%20432%20002> <tel:+48791432002 <+48%20791%20432%20002>
> >> <tel:+48%20791%20432%20002>>
> >> >> E: michal.walenia@polidea.com
> >> <ma...@polidea.com>
> >> <mailto:michal.walenia@polidea.com
> >> <ma...@polidea.com>>
> >> >>
> >> >> Unique Tech
> >> >> Check out our projects! <https://www.polidea.com/our-work>
> >> >>
> >>
>