You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by Helge Pfeiffer <ro...@itu.dk> on 2021/06/15 12:09:30 UTC
The Impact of SonarCloud on Defect Reports?
Dear Ratis developers,
My name is Helge, I am a researcher at IT University of Copenhagen [1].
I am currently conducting a study on the impact of continuous code quality assessment tools (SonarQube) on defects.
I am writing to you -the Ratis developers-, since I found that Ratis is the only ASF project, for which the amount of reported defects, i.e., bugs in Jira, are significantly reduced after SonarCloud was introduced to development.
The first SonarCloud assessment of Ratis is from 2020-05-28 [2].
In Jira, on average 4 defect reports per week (median 3) in the year before the first SonarCloud assessment drop to on average 2 defect reports per week (median 2) in the year after. That is impressive. I would like to understand if the reduction of defect reports is caused by introduction of SonarCloud to development or if this is just a coincidence.
Therefore, I have some questions for which I would really appreciate your feedback:
1) Do you -the Ratis developers- believe that the reduction of defect reports on Jira is actually related to your work on remediation of SonarCloud issues (code smells, vulnerabilities, or bugs)?
1.2) If not, what do you believe is the reason for decreasing defect reports?
2) Is it correct that the Ratis project did not use SonarQube (the one hosted by the Apache Infrastructure Team) before using SonarCloud?
3) SonarCloud issues (code smells, vulnerabilities, or bugs) are generally decreasing (https://sonarcloud.io/project/activity?id=apache-ratis). Is that due to:
3.1) Your active work on remedying these SonarCloud issues?
3.2) Changes of the applied SonarQube quality profile?
3.3) A change of your coding to respect SonarQube rules proactively?
Thank you in advance for your feedback and consideration. I will share the results of my work with you as soon they are written down in a presentable format. Below, I present preliminary insights of my current analysis.
Best regards,
Helge
## Some insights...
I find 16 Jira issues that mention `[Ss]onar` and that are resolved[3]. For these, I find 15 commits[4] that address and reference these Jira issues. In between the first SonarCloud assessment (2020-05-28) and yesterday (2021-06-14) there are 315 commits to the repositories main branch (up to version [https://github.com/apache/ratis/commit/a7315511bddcc12dc4acea25f2d0d71608a8591c](a73155)). That is, 15 out of 315 commits (ca. 5%) are related to SonarCloud and its code quality assessments.
I am hesitant to believe that that these 15 commits can have such a strong impact to the decreasing amount of defect reports in Jira, in particular since four of them are about configuring and setting-up SonarCloud.
Also, I believe that these commits cannot explain the generally decreasing trends of code smells, bugs, and vulnerabilities in SonarCloud's dashboard[5]. Especially, since Ratis is growing with 9905 lines from 2020-05-28 to 2021-06-14 and more lines in software usually mean higher chance of introduction of defects and violation of SonarCloud issues.
I find no commits that mention code `[Ss]mell`s or `[Vv]ulnerabilities` in the period from 2020-05-28 to 2021-06-14. That is, I believe there are no commits that address SonarCloud code smells or vulnerabilities. From 2020-05-28 to 2021-06-14, I find 10 commits that mention the term `[Bb]ug`[6]. After inspecting all of these and the respective Jira issues, I believe that they do not fix bugs that are reported by SonarCloud but bugs in the sense of defects.
-------------------
[1] https://www.itu.dk/people/ropf/ and https://www.researchgate.net/profile/Helge-Pfeiffer-2
[2] https://sonarcloud.io/project/activity?id=apache-ratis&selected_date=2020-05-28T10%3A30%3A20%2B0000
[3] Resolved issues that mention `[Ss]onar` or that are a sub-task of one of these (ordered by Jira ID):
* https://issues.apache.org/jira/browse/RATIS-940
* https://issues.apache.org/jira/browse/RATIS-948
* https://issues.apache.org/jira/browse/RATIS-950
* https://issues.apache.org/jira/browse/RATIS-953
* https://issues.apache.org/jira/browse/RATIS-1051
* https://issues.apache.org/jira/browse/RATIS-1052
* https://issues.apache.org/jira/browse/RATIS-1053
* https://issues.apache.org/jira/browse/RATIS-1054
* https://issues.apache.org/jira/browse/RATIS-1055
* https://issues.apache.org/jira/browse/RATIS-1075
* https://issues.apache.org/jira/browse/RATIS-1306
* https://issues.apache.org/jira/browse/RATIS-1311
* https://issues.apache.org/jira/browse/RATIS-1364
* https://issues.apache.org/jira/browse/RATIS-1365
* https://issues.apache.org/jira/browse/RATIS-1366
* https://issues.apache.org/jira/browse/RATIS-1367
[4] Commits that refer to a resolved `[Ss]onar` related issue:
* RATIS-940: https://github.com/apache/ratis/commit/00f1747a1915f42fd256e9a5b7b8e0131cd06c2c
* RATIS-948: https://github.com/apache/ratis/commit/a2f3895396a81ee6e31d6fd1a8a6c8a7bf121dd6
* RATIS-950: https://github.com/apache/ratis/commit/43a042a8bbe123bcb5e567af0aeced12eb299290
* RATIS-953: https://github.com/apache/ratis/commit/02caace296f4414de3eda9f4469dbd806ca594b1
* RATIS-1075: https://github.com/apache/ratis/commit/9b1d2c18f5677f3d443344e69f98e6c3ac953835
* RATIS-1051: https://github.com/apache/ratis/commit/cf9ed0864615e82083f6be5f4f958563dbf242ad
* RATIS-1052: https://github.com/apache/ratis/commit/0d964950e8fdf6887e53e8eb53695a14d29314c9
* RATIS-1052: https://github.com/apache/ratis/commit/6dd51454994a843fde10933d2ec3daedd6fb36de
* RATIS-1055: https://github.com/apache/ratis/commit/61f038b6bea2934e910e8504df0d8c79d6d34799
* RATIS-1306: https://github.com/apache/ratis/commit/9d1b711f9d4606145e19a28270b0c50b04dc869b
* RATIS-1311: https://github.com/apache/ratis/commit/87bd1fd1df9f02e83b973291d507ad002bd9d3f4
* RATIS-1364: https://github.com/apache/ratis/commit/9577d564eeac36cf449843d34b7010b33d634818
* RATIS-1365: https://github.com/apache/ratis/commit/ff8aa668f1a0569ba5e6b0f30dbd51a673913344
* RATIS-1366: https://github.com/apache/ratis/commit/d65ca26a0291fc6067f860eff4ff3092d25c0aec
* RATIS-1367: https://github.com/apache/ratis/commit/040bc52e19a5e36f5710ccd4fc1981e862e691e8
[5] https://sonarcloud.io/project/activity?id=apache-ratis
[6] Commits that mention `[Bb]ug`s in the period 2020-05-28 to 2021-06-14:
* https://github.com/apache/ratis/commit/af358415cc8ebc70665d73e5fc4812b7cb669c75
* https://github.com/apache/ratis/commit/32016e4a40ae44bb3c3718880327bfb085bff509
* https://github.com/apache/ratis/commit/86dd7fa68081e33428ed65dfdb613032b1378560
* https://github.com/apache/ratis/commit/e5a052c18766293b94cc89e3a34a469193009537
* https://github.com/apache/ratis/commit/79223f89054109d2c499d81a149e60c15fc90453
* https://github.com/apache/ratis/commit/17a2198b0016a66494da4344d218593ed2c4d7de
* https://github.com/apache/ratis/commit/f51196455679443be438a1ddb04bd6860f4ad6dc
* https://github.com/apache/ratis/commit/cfa6c4f70e8dc296c921b7c8b33511d58270ab39
* https://github.com/apache/ratis/commit/195c572024fc5fd88d00c3c0985c01215d2719aa
* https://github.com/apache/ratis/commit/9fc6a1163b993177fd884bba4212326bb6abbf73