Disabling, circumventing and altering CI checks

[Marco de Abreu <ma...@apache.org>]

Hello,

recently I have noticed a trend of CI checks being disabled, circumvented
or altered as part of a Pull Request. While I understand that these systems
might decline your PR, they execute checks and enforce standards that the
community has agreed on. The reason for these standards is not to give
contributors a hard time but instead to improve the maintainability,
compatibility and stability of our code base.

I acknowledge that there might have been a few issues recently that
resulted in an increased rate of false errors and I agree that this is not
ideal. But none the less, I'd like to encourage everybody to think about
the long-term impact of tampering with these systems.

Since I prefer to not call out names, I will keep this email generic. But a
few examples of what such a problematic change could look like include:
- Changing Jenkinsfiles
- Changing Dockerfiles
- Changing runtime functions
- Disabling linting or adding exclusions
- Adding additional preprocessor statements that avoid that codepath from
being checked by CI
- Disabling tests or making them conditional to avoid execution by CI

While I understand that new contributors might not be that used to the dos
and donts of a software project, I'd like to remind the committers of this
project to think in the best interest of the project and adhere to
engineering best practices. While I understand that there might be external
factors that create the urge to take these shortcuts, it's the committers
responsibility to ensure that they don't make their way into the codebase
or get remedied as soon as possible.

Thanks for your understanding.

Best regards,
Marco

Re: Disabling, circumventing and altering CI checks

[Mu Li <li...@gmail.com>]

Hi Marco,

Thank you for helping CI all the time. You did an incredible job on it. 

Please let me explain why it’s urgent we need to update our CI to allow fast developing. 

In this summer, we managed to hire a large amount of interns to help, they did great to contribute to MXNet. But CI is one of the blocking issues. You can see that just for numpy 70+ PRs are open.  VP in your org Our engineering VP expressed his concern about the numpy project progress. I heard you are switching to new role in another org, so it may not matter you too much. But it does affect the resource I can have for this project.

Conservatism is not an option. MXNet has 10x less PR merged per day compared Pytorch and Tensorflow. MXNet’s downloads are also 10x less. We need to address it quickly. A large amount of contributors are paid to work on MXNet. These companies may withdraw their resources if MXNet is not so successful. We need to continue the PR boom for the numpy project.

I saw you are already work on it. But we should be open for other changes. We should optimize our test. I wrote most of the test before 2016 and the original Jenkinsfile. I knew there are a large amount of redundant tests. We can resume some of them. Also some operators are tested with a large amount of random inputs, we can also reduce them. Eg if an op is tested by 1m iterations, we can reduce it to 100k with a changing random seed, so we can still test 1m times with 10 tests. 

There are a large amount of best practices in software testing we can follow. I hope the community is open for changes, for both short term fixes and long term improvements.


Best
Mu

> On Aug 24, 2019, at 1:01 AM, Marco de Abreu <ma...@apache.org> wrote:
> 
> Hello,
> 
> recently I have noticed a trend of CI checks being disabled, circumvented
> or altered as part of a Pull Request. While I understand that these systems
> might decline your PR, they execute checks and enforce standards that the
> community has agreed on. The reason for these standards is not to give
> contributors a hard time but instead to improve the maintainability,
> compatibility and stability of our code base.
> 
> I acknowledge that there might have been a few issues recently that
> resulted in an increased rate of false errors and I agree that this is not
> ideal. But none the less, I'd like to encourage everybody to think about
> the long-term impact of tampering with these systems.
> 
> Since I prefer to not call out names, I will keep this email generic. But a
> few examples of what such a problematic change could look like include:
> - Changing Jenkinsfiles
> - Changing Dockerfiles
> - Changing runtime functions
> - Disabling linting or adding exclusions
> - Adding additional preprocessor statements that avoid that codepath from
> being checked by CI
> - Disabling tests or making them conditional to avoid execution by CI
> 
> While I understand that new contributors might not be that used to the dos
> and donts of a software project, I'd like to remind the committers of this
> project to think in the best interest of the project and adhere to
> engineering best practices. While I understand that there might be external
> factors that create the urge to take these shortcuts, it's the committers
> responsibility to ensure that they don't make their way into the codebase
> or get remedied as soon as possible.
> 
> Thanks for your understanding.
> 
> Best regards,
> Marco