[DISCUSS] Use more strict revert/merge rules

["Elek, Marton" <el...@apache.org>]

During the last 5 weeks (since the separation of hadoop-ozone 
repository) about 177 commits has been added to the master branch of Ozone.

This means ~7 commits per working day.

With this development velocity it's very important to keep the master 
branch as stable as possible.

Any potential CI failure can hide additional problems which makes either 
harder or more risky to commit/merge any PR. Multiple new test failures 
are introduced because the problem was not visible due to an other, 
earlier problem.

I think we need to do additional steps to keep the master branch more 
stable. I think there are two main approaches.


  1. Revert immediately the commits after any kind of failures.

  2. Merge to the master with a script only if CI build passes


There are multiple options to implement 2nd: for example after the 
review it would be enough to add a "ready-to-merge" label and an 
additional job would execute all the tests and merge the PR to the 
master only if all the tests are passed.

This second approach has some hard limitations:

  * Commits can't be signed by the committer if the merge is organized 
by a script.
  * It requires tricky github actions logic as ASF INFRA doesn't allow 
to use any external services which require write permission to the repo 
(such as mergify)

THEREFORE I PROPOSE to follow the 1st approach:

  1. Revert any commit IMMEDIATELY if new failures are introduced. (And 
communicate everywhere that we need all of the new commits and potential 
reverts are not against the code, just an integration step for the 
stability.)

  2. In case of a flaky test where the source of the flakiness is 
unknown, ignore it immediately with a commit (without specific review!). 
The @Ignore annotation should include the JIRA where the problem is 
described/tracked.


What do you think?


Marton

PS: Right now both acceptance tests and unit tests are in bad shape. 
Acceptance tests were broken by a commit two weeks ago, unit tests were 
broken by a commit 4 days ago. In both cases the problem was not fully 
visible due to other existing problems.

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-dev-help@hadoop.apache.org