You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/04/09 12:57:48 UTC
[GitHub] [ignite-teamcity-bot] glukos opened a new pull request #159: Show
tests on TC board that were stable and have become flaky
glukos opened a new pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406255215
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/board/BoardService.java
##########
@@ -79,12 +81,51 @@
* @param creds Credentials.
*/
public BoardSummaryUi summary(ICredentialsProv creds) {
- issuesToDefectsLater();
+ issuesToDefectsLater(creds);
Map<Integer, Future<FatBuildCompacted>> allBuildsMap = new HashMap<>();
+
+
List<DefectCompacted> defects = defectStorage.loadAllDefects();
+ for (DefectCompacted next : defects) {
+
+ String srvCode = next.tcSrvCode(compactor);
+
+ ITeamcityIgnited tcIgnited = tcProv.server(srvCode, creds);
+
+ if (!creds.hasAccess(srvCode))
+ continue;
+
+ Map<Integer, DefectFirstBuild> build = next.buildsInvolved();
+ for (DefectFirstBuild cause : build.values()) {
+ FatBuildCompacted firstBuild = cause.build();
+
+ int projectId = firstBuild.buildTypeId();
+ int branchName = firstBuild.branchName();
+
+ for (DefectIssue issue : cause.issues()) {
+ int fullSuiteNameAndFullTestName = issue.testNameCid();
+
+ IRunHistory runStat = tcIgnited.getTestRunHist(fullSuiteNameAndFullTestName, projectId, branchName);
+
+ if (runStat == null)
+ continue;
+
+ Integer firstFailedBuildId = runStat.detectTemplate(EventTemplates.stablePassedTest);
+
+ boolean isNewTestWithHighFlakyRate = IssueType.newTestWithHighFlakyRate.code().equals(compactor.getStringFromId(issue.issueTypeCode()));
+
+ if (isNewTestWithHighFlakyRate && firstFailedBuildId != null)
+ resolveDefect(next.id(), creds, false);
Review comment:
We shouldn't actually "resolve" defect here. It will disappear from the board in such case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406246602
##########
File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/issue/IssueDetector.java
##########
@@ -469,10 +469,14 @@ private boolean registerTestFailIssues(ITeamcityIgnited tcIgnited,
if (firstFailedBuildId != null) {
type = IssueType.newFailure;
+
final String flakyComments = runStat.getFlakyComments();
if (!Strings.isNullOrEmpty(flakyComments)) {
- if (runStat.detectTemplate(EventTemplates.newFailureForFlakyTest) == null) {
+ if (runStat.getFlakyRate() > cfg.flakyRate() &&
Review comment:
Do we check flaky rate for all time here?
Does it mean that if test has history of 500 stable runs and then gets flaky with rate 30% we'll detect only after ~1000 runs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
` int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406255746
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/board/BoardService.java
##########
@@ -79,12 +81,51 @@
* @param creds Credentials.
*/
public BoardSummaryUi summary(ICredentialsProv creds) {
- issuesToDefectsLater();
+ issuesToDefectsLater(creds);
Map<Integer, Future<FatBuildCompacted>> allBuildsMap = new HashMap<>();
+
+
List<DefectCompacted> defects = defectStorage.loadAllDefects();
+ for (DefectCompacted next : defects) {
+
+ String srvCode = next.tcSrvCode(compactor);
+
+ ITeamcityIgnited tcIgnited = tcProv.server(srvCode, creds);
+
+ if (!creds.hasAccess(srvCode))
+ continue;
+
+ Map<Integer, DefectFirstBuild> build = next.buildsInvolved();
+ for (DefectFirstBuild cause : build.values()) {
+ FatBuildCompacted firstBuild = cause.build();
+
+ int projectId = firstBuild.buildTypeId();
+ int branchName = firstBuild.branchName();
+
+ for (DefectIssue issue : cause.issues()) {
+ int fullSuiteNameAndFullTestName = issue.testNameCid();
+
+ IRunHistory runStat = tcIgnited.getTestRunHist(fullSuiteNameAndFullTestName, projectId, branchName);
+
+ if (runStat == null)
+ continue;
+
+ Integer firstFailedBuildId = runStat.detectTemplate(EventTemplates.stablePassedTest);
+
+ boolean isNewTestWithHighFlakyRate = IssueType.newTestWithHighFlakyRate.code().equals(compactor.getStringFromId(issue.issueTypeCode()));
+
+ if (isNewTestWithHighFlakyRate && firstFailedBuildId != null)
+ resolveDefect(next.id(), creds, false);
Review comment:
It should be marked as "green" instead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252012
##########
File path: tcbot-teamcity-ignited/src/main/java/org/apache/ignite/tcignited/history/AbstractRunHist.java
##########
@@ -74,6 +74,11 @@
"changed its status [" + statusChange + "/" + getInvocations().count() + "] without code modifications";
}
+ /** */
+ @Override public float getFlakyRate() {
+ return (float) getStatusChangesWithoutCodeModification() / getRunsCount() * 100;
Review comment:
We may underestimate flaky rate here.
Every status change with unrelated code modifications won't count.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] sergeyuttsel commented on a change in pull
request #159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
sergeyuttsel commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r408239605
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
I removed new event template.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r409693850
##########
File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/issue/IssueDetector.java
##########
@@ -484,6 +493,28 @@ private boolean registerTestFailIssues(ITeamcityIgnited tcIgnited,
}
}
+ double flakyRate = 0;
+
+ if (firstFailedBuildId == null || type == null) {
+ List<Invocation> invocations = runStat.getInvocations().
+ filter(invocation -> invocation != null && invocation.status() != InvocationData.MISSING)
+ .collect(Collectors.toList());
+
+ int okTestsRow = (int)Math.ceil(Math.log(1 - cfg.confidence()) / Math.log(1 - cfg.flakyRate() / 100.0));
+
+ if (invocations.size() >= okTestsRow) {
+ List<Invocation> lastInvocations = invocations.subList(invocations.size() - okTestsRow, invocations.size());
Review comment:
Are we protected from ArrayIndexOutOfBounds here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] sergeyuttsel commented on a change in pull
request #159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
sergeyuttsel commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r410078566
##########
File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/issue/IssueDetector.java
##########
@@ -484,6 +493,28 @@ private boolean registerTestFailIssues(ITeamcityIgnited tcIgnited,
}
}
+ double flakyRate = 0;
+
+ if (firstFailedBuildId == null || type == null) {
+ List<Invocation> invocations = runStat.getInvocations().
+ filter(invocation -> invocation != null && invocation.status() != InvocationData.MISSING)
+ .collect(Collectors.toList());
+
+ int okTestsRow = (int)Math.ceil(Math.log(1 - cfg.confidence()) / Math.log(1 - cfg.flakyRate() / 100.0));
+
+ if (invocations.size() >= okTestsRow) {
+ List<Invocation> lastInvocations = invocations.subList(invocations.size() - okTestsRow, invocations.size());
Review comment:
We protected by checking `invocations.size() >= okTestsRow`. The method `subList` include a "from" element and exclude "to" one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406257405
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/board/BoardService.java
##########
@@ -79,12 +81,51 @@
* @param creds Credentials.
*/
public BoardSummaryUi summary(ICredentialsProv creds) {
- issuesToDefectsLater();
+ issuesToDefectsLater(creds);
Map<Integer, Future<FatBuildCompacted>> allBuildsMap = new HashMap<>();
+
+
List<DefectCompacted> defects = defectStorage.loadAllDefects();
+ for (DefectCompacted next : defects) {
+
+ String srvCode = next.tcSrvCode(compactor);
+
+ ITeamcityIgnited tcIgnited = tcProv.server(srvCode, creds);
+
+ if (!creds.hasAccess(srvCode))
+ continue;
+
+ Map<Integer, DefectFirstBuild> build = next.buildsInvolved();
+ for (DefectFirstBuild cause : build.values()) {
+ FatBuildCompacted firstBuild = cause.build();
+
+ int projectId = firstBuild.buildTypeId();
+ int branchName = firstBuild.branchName();
+
+ for (DefectIssue issue : cause.issues()) {
+ int fullSuiteNameAndFullTestName = issue.testNameCid();
+
+ IRunHistory runStat = tcIgnited.getTestRunHist(fullSuiteNameAndFullTestName, projectId, branchName);
+
+ if (runStat == null)
+ continue;
+
+ Integer firstFailedBuildId = runStat.detectTemplate(EventTemplates.stablePassedTest);
+
+ boolean isNewTestWithHighFlakyRate = IssueType.newTestWithHighFlakyRate.code().equals(compactor.getStringFromId(issue.issueTypeCode()));
+
+ if (isNewTestWithHighFlakyRate && firstFailedBuildId != null)
+ resolveDefect(next.id(), creds, false);
Review comment:
We should assign status "IssueResolveStatus.FIXED" to the defect, probably better place to do it is BoardService#processIssue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
` int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on issue #159: Show tests on
TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on issue #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#issuecomment-611514214
@dspavlov Can you please take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406188326
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
Review comment:
We need to calculate length of success chain from flakyRate.
Formula is ln(1-confidence [95%]) / ln (1-flakyRate).
It would be better to extract confidence to config as well.
Other option is to extract chain length explicitly [13].
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406261962
##########
File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/issue/IssueDetector.java
##########
@@ -469,10 +469,14 @@ private boolean registerTestFailIssues(ITeamcityIgnited tcIgnited,
if (firstFailedBuildId != null) {
type = IssueType.newFailure;
+
final String flakyComments = runStat.getFlakyComments();
if (!Strings.isNullOrEmpty(flakyComments)) {
- if (runStat.detectTemplate(EventTemplates.newFailureForFlakyTest) == null) {
+ if (runStat.getFlakyRate() > cfg.flakyRate() &&
Review comment:
The other problem is that literally EVERY flaky test is applicable to this criteria and will appear on the board.
We have to display flaky test that previously were stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252012
##########
File path: tcbot-teamcity-ignited/src/main/java/org/apache/ignite/tcignited/history/AbstractRunHist.java
##########
@@ -74,6 +74,11 @@
"changed its status [" + statusChange + "/" + getInvocations().count() + "] without code modifications";
}
+ /** */
+ @Override public float getFlakyRate() {
+ return (float) getStatusChangesWithoutCodeModification() / getRunsCount() * 100;
Review comment:
We may underestimate flaky rate here.
Every status change with unrelated code modifications won't count,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite-teamcity-bot] glukos commented on a change in pull request
#159: Show tests on TC board that were stable and have become flaky
Posted by GitBox <gi...@apache.org>.
glukos commented on a change in pull request #159: Show tests on TC board that were stable and have become flaky
URL: https://github.com/apache/ignite-teamcity-bot/pull/159#discussion_r406252751
##########
File path: tcbot-engine/src/main/java/org/apache/ignite/tcbot/engine/issue/EventTemplates.java
##########
@@ -53,6 +55,11 @@
FAIL, FAIL}
);
+ public static final EventTemplate stablePassedTest = new EventTemplate(
+ new int[]{OK_OR_FAILURE},
+ new int[]{OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK, OK}
+ );
+
Review comment:
Should we add new event template to public static ArrayList<EventTemplate> templates; ?
According to the its only usage we shouldn't:
`
int buildsToQry = EventTemplates.templates.stream().mapToInt(EventTemplate::cntEvents).max().getAsInt();
ITcBotUserCreds creds = Preconditions.checkNotNull(backgroundOpsCreds, "Server should be authorized");
tbProc.getTrackedBranchTestFailures(
brachName,
false,
buildsToQry,
creds,
SyncMode.RELOAD_QUEUED,
false,
null,
null,
DisplayMode.None,
null,
-1, false, false);`
Should we rename field "templates" in this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services