You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Tyson Hamilton <ty...@google.com> on 2020/05/21 23:12:54 UTC

Apache Beam Fixit 2020-06-08

Why

Google has a Fixit week to improve Engineering Excellence and we want to
use it to improve Apache Beam!
What

For the week of 2020-06-08 you will see an increased focus from Google
engineers on improving Engineering Excellence. We’re asking the community
to join in and commit as much, or as little, as you’re able to -- anything
helps!

Goals:

   -

   Reduce flakiness
   <https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20>
   [1] in tests across unit/integration tests in Jenkins.
   -

   Reduce overall Jenkins usage.
   -

   Improve project health signals: add signal, fix signal, split up jobs
   for clearer signals.
   -

      See post-commit greenness
      <http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s>
      [2], coverage
      <https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22>
      [3], precommit latency
      <http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1>
      [4].


Examples

   -

   triaging @Ignored tests and possibly removing decrepit ones
   <https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E>
   [5]
   -

   [BEAM-9999] Remove support for EOLed runners (Apex, etc.)
   <https://issues.apache.org/jira/browse/BEAM-9999> [6]
   -

   [BEAM-2762] Coverage report for Python code
   <https://issues.apache.org/jira/browse/BEAM-2762> [7]
   -

   [BEAM-1399] Coverage numbers are not accurate
   <https://issues.apache.org/jira/browse/BEAM-1399> [8]
   -

   Jira issues for test failures
   <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
   [9]


How

   -

   Check the ‘beam-fixit’
   <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
   [10] Jira label for open issues or see the suggestions above.
   -

      Ensure the issue you’re working on, or completed, is on the beam-fixit
      <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
      Jira label.
      -

   Assign an issue to yourself and FIXIT!


Text Links:

[1]:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20

[2]:
http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s

[3]:
https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22

[4]:
http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1

[5]:
https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E

[6]: https://issues.apache.org/jira/browse/BEAM-9999

[7]: https://issues.apache.org/jira/browse/BEAM-2762

[8]: https://issues.apache.org/jira/browse/BEAM-1399

[9]:
https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

[10]:
https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit

Best,
-Tyson

Re: Apache Beam Fixit 2020-06-08

Posted by Kenneth Knowles <ke...@apache.org>.

This is the perfect time for improving engineering excellence. Reliability
is especially important right now.

Improved greenness, reduced flakiness, improved coverage (and highlighting
the coverage signal), triaging ignored tests removing dead code. These are
all clear wins for any project.

For Beam Java I might also add trying to (re-)incorporate lightweight
analyses that significantly improve code quality and reliability, such as:

 - nullness checking (got broken/lost when we moved to spotbugs, which
seems to not really check nearly as much as findbugs)
 - IWYU analysis (got lost when we moved to Gradle; now available but not
turned on for most modules)
 - dependency convergence
 - eliminating rawtypes, unchecked casts, and other legacy Java patterns
(will need to turn on -Werror for them on a per-module basis)

Kenn

On Thu, May 21, 2020 at 5:21 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> Thanks, Tyson.
>
> I noticed a few links could be corrected, fixes inline.
>
> On Thu, May 21, 2020 at 4:13 PM Tyson Hamilton <ty...@google.com> wrote:
>
>> Why
>>
>> Google has a Fixit week to improve Engineering Excellence and we want to
>> use it to improve Apache Beam!
>> What
>>
>> For the week of 2020-06-08 you will see an increased focus from Google
>> engineers on improving Engineering Excellence. We’re asking the community
>> to join in and commit as much, or as little, as you’re able to -- anything
>> helps!
>>
>> Goals:
>>
>>    -
>>
>>    Reduce flakiness
>>    <https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20>
>>    [1] in tests across unit/integration tests in Jenkins.
>>    -
>>
>>    Reduce overall Jenkins usage.
>>    -
>>
>>    Improve project health signals: add signal, fix signal, split up jobs
>>    for clearer signals.
>>    -
>>
>>       See post-commit greenness
>>       <http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s>
>>       [2], coverage
>>       <https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22>
>>       [3], precommit latency
>>       <http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1>
>>       [4].
>>
>>
>> Examples
>>
>>    -
>>
>>    triaging @Ignored tests and possibly removing decrepit ones
>>    <https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E>
>>    [5]
>>    -
>>
>>    [BEAM-9999] Remove support for EOLed runners (Apex, etc.)
>>    <https://issues.apache.org/jira/browse/BEAM-9999> [6]
>>    -
>>
>>    [BEAM-2762] Coverage report for Python code
>>    <https://issues.apache.org/jira/browse/BEAM-2762> [7]
>>    -
>>
>>    [BEAM-1399] Coverage numbers are not accurate
>>    <https://issues.apache.org/jira/browse/BEAM-1399> [8]
>>    -
>>
>>    Jira issues for test failures
>>    <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>>    [9]
>>
>> Jira issues for test failures link
> <https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
> .
>
>>
>> How
>>
>>    -
>>
>>    Check the ‘beam-fixit’
>>    <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
>>    [10] Jira label for open issues or see the suggestions above.
>>
>> Jira issues with beam-fixit label link
> <https://issues.apache.org/jira/issues/?jql=labels%20%3D%20beam-fixit>.
>
>
>>    -
>>
>>       Ensure the issue you’re working on, or completed, is on the
>>       beam-fixit
>>       <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
>>       Jira label.
>>       -
>>
>>    Assign an issue to yourself and FIXIT!
>>
>>
>> Text Links:
>>
>> [1]:
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20
>>
>> [2]:
>> http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s
>>
>> [3]:
>> https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22
>>
>> [4]:
>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>
>> [5]:
>> https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E
>>
>> [6]: https://issues.apache.org/jira/browse/BEAM-9999
>>
>> [7]: https://issues.apache.org/jira/browse/BEAM-2762
>>
>> [8]: https://issues.apache.org/jira/browse/BEAM-1399
>>
>> [9]:
>> https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-
>> <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>>
>>
> failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>> <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>>
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
>
>> [10]:
>> https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit
>>
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%20beam-fixit
>
>>
>> Best,
>> -Tyson
>>
>

Re: Apache Beam Fixit 2020-06-08

Posted by Valentyn Tymofieiev <va...@google.com>.

Thanks, Tyson.

I noticed a few links could be corrected, fixes inline.

On Thu, May 21, 2020 at 4:13 PM Tyson Hamilton <ty...@google.com> wrote:

> Why
>
> Google has a Fixit week to improve Engineering Excellence and we want to
> use it to improve Apache Beam!
> What
>
> For the week of 2020-06-08 you will see an increased focus from Google
> engineers on improving Engineering Excellence. We’re asking the community
> to join in and commit as much, or as little, as you’re able to -- anything
> helps!
>
> Goals:
>
>    -
>
>    Reduce flakiness
>    <https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20>
>    [1] in tests across unit/integration tests in Jenkins.
>    -
>
>    Reduce overall Jenkins usage.
>    -
>
>    Improve project health signals: add signal, fix signal, split up jobs
>    for clearer signals.
>    -
>
>       See post-commit greenness
>       <http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s>
>       [2], coverage
>       <https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22>
>       [3], precommit latency
>       <http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1>
>       [4].
>
>
> Examples
>
>    -
>
>    triaging @Ignored tests and possibly removing decrepit ones
>    <https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E>
>    [5]
>    -
>
>    [BEAM-9999] Remove support for EOLed runners (Apex, etc.)
>    <https://issues.apache.org/jira/browse/BEAM-9999> [6]
>    -
>
>    [BEAM-2762] Coverage report for Python code
>    <https://issues.apache.org/jira/browse/BEAM-2762> [7]
>    -
>
>    [BEAM-1399] Coverage numbers are not accurate
>    <https://issues.apache.org/jira/browse/BEAM-1399> [8]
>    -
>
>    Jira issues for test failures
>    <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>    [9]
>
> Jira issues for test failures link
<https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
.

>
> How
>
>    -
>
>    Check the ‘beam-fixit’
>    <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
>    [10] Jira label for open issues or see the suggestions above.
>
> Jira issues with beam-fixit label link
<https://issues.apache.org/jira/issues/?jql=labels%20%3D%20beam-fixit>.


>    -
>
>       Ensure the issue you’re working on, or completed, is on the
>       beam-fixit
>       <https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit>
>       Jira label.
>       -
>
>    Assign an issue to yourself and FIXIT!
>
>
> Text Links:
>
> [1]:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20
>
> [2]:
> http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1&refresh=30s
>
> [3]:
> https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22
>
> [4]:
> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>
> [5]:
> https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E
>
> [6]: https://issues.apache.org/jira/browse/BEAM-9999
>
> [7]: https://issues.apache.org/jira/browse/BEAM-2762
>
> [8]: https://issues.apache.org/jira/browse/BEAM-1399
>
> [9]:
> https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-
> <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>
>
failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> <https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC>
>
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC


> [10]:
> https://issues.apache.org/jira/browse/BEAM-9999?jql=labels%20%3D%20beam-fixit
>
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20beam-fixit

>
> Best,
> -Tyson
>