You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@netbeans.apache.org by "mbien (via GitHub)" <gi...@apache.org> on 2025/01/23 22:32:28 UTC
[I] Improve test reliability [netbeans]
mbien opened a new issue, #8183:
URL: https://github.com/apache/netbeans/issues/8183
### Body
tests are still failing too often. Since we don't have good stats, I tried to crawl workflow results using a rudimentary [bash script](https://gist.github.com/mbien/6d5f82f6fbb6c67bfbb702bf91fe3c0c) calling into the github API.
The script scans workflow runs and filters merges to master which failed, then scans re-runs too and prints the failed steps. Rest is just `sort` and `uniq`.
```
14 "ide/project.ant"
13 "java.hints batch1"
11 "java.file.launcher"
10 "ide/css.editor"
7 "groovy/groovy.editor"
6 "webcommon/html.knockout"
3 "platform/o.n.bootstrap"
2 "java/maven"
2 "java.hints batch2"
1 "web.beans"
1 "platform/openide.nodes"
1 "java.editor"
```
note: this does not take the retry script into account which might be active on [some steps](https://github.com/apache/netbeans/blob/cbfd04788a960b293b2b16521629aabdafee1e84/.github/workflows/main.yml#L263)
here the results with the links pointing directly to the workflow run:
<details>
run: https://github.com/apache/netbeans/actions/runs/12936028241 (4 attempts)
- `groovy/groovy.editor`
- `groovy/groovy.editor`
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12934523662 (2 attempts)
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12933263724 (3 attempts)
- `ide/project.ant`
- `ide/project.ant`
- `java.editor`
- `java/maven`
run: https://github.com/apache/netbeans/actions/runs/12933078681 (4 attempts)
- `java.hints batch1`
- `java.hints batch2`
- `platform/o.n.bootstrap`
- `java.hints batch1`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12912321783 (3 attempts)
- `java.file.launcher`
- `java.hints batch1`
- `platform/o.n.bootstrap`
- `ide/css.editor`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12872488235 (3 attempts)
- `webcommon/html.knockout`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12869051716 (3 attempts)
- `java.hints batch1`
- `ide/css.editor`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12857668868 (2 attempts)
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12851818621 (3 attempts)
- `ide/css.editor`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12846158264 (2 attempts)
- `ide/project.ant`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12836525165 (2 attempts)
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12836146985 (2 attempts)
- `java/maven`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12826345964 (3 attempts)
- `ide/project.ant`
- `platform/o.n.bootstrap`
- `webcommon/html.knockout`
- `ide/project.ant`
run: https://github.com/apache/netbeans/actions/runs/12825920011 (3 attempts)
- `ide/project.ant`
- `groovy/groovy.editor`
- `ide/project.ant`
run: https://github.com/apache/netbeans/actions/runs/12825908774 (2 attempts)
- `ide/project.ant`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12795877604 (2 attempts)
- `ide/project.ant`
- `java.hints batch1`
- `java.file.launcher`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12794975127 (2 attempts)
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12791936642 (2 attempts)
- `ide/project.ant`
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12765772367 (2 attempts)
- `ide/project.ant`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12764904231 (2 attempts)
- `ide/project.ant`
- `java.file.launcher`
- `platform/openide.nodes`
run: https://github.com/apache/netbeans/actions/runs/12762278637 (3 attempts)
- `java.file.launcher`
- `ide/css.editor`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12762263024 (3 attempts)
- `java.hints batch1`
- `java.file.launcher`
- `java.file.launcher`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12749609504 (2 attempts)
- `java.hints batch1`
- `java.hints batch2`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12722000825 (4 attempts)
- `ide/project.ant`
- `web.beans`
- `groovy/groovy.editor`
- `java.hints batch1`
- `ide/css.editor`
- `java.hints batch1`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12721993365 (2 attempts)
- `ide/project.ant`
- `webcommon/html.knockout`
</details>
### Committer
- [x] I acknowledge that I am a maintainer/committer in the Apache NetBeans project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2614055656
for `java.file.launcher` -> https://github.com/apache/netbeans/pull/8193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2613004149
i take a look at `ide/project.ant` (testGetBuildScriptState: org.netbeans.spi.project.support.ant.GeneratedFilesHelperTest)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-5533384462
generated the list again:
<details>
run: https://github.com/apache/netbeans/actions/runs/33672678804/attempts/1 (5 attempts)
"java.source.base"
"java.source.base"
"java.source.base"
"java.source.base"
run: https://github.com/apache/netbeans/actions/runs/33653847307/attempts/1 (2 attempts)
"java.source.base"
run: https://github.com/apache/netbeans/actions/runs/33591884871/attempts/1 (2 attempts)
"micronaut"
run: https://github.com/apache/netbeans/actions/runs/33325915577/attempts/1 (2 attempts)
"java.source.base"
"java/maven"
run: https://github.com/apache/netbeans/actions/runs/33129395096/attempts/1 (3 attempts)
"java.source.base"
"java.source.base"
run: https://github.com/apache/netbeans/actions/runs/33014674239/attempts/1 (2 attempts)
"apisupport.ant"
"refactoring.java"
run: https://github.com/apache/netbeans/actions/runs/33004304103/attempts/1 (2 attempts)
"apisupport.ant"
run: https://github.com/apache/netbeans/actions/runs/32884389025/attempts/1 (2 attempts)
"java/java.lsp.server"
run: https://github.com/apache/netbeans/actions/runs/31745293654/attempts/1 (2 attempts)
"java.hints batch1"
run: https://github.com/apache/netbeans/actions/runs/31737988936/attempts/1 (2 attempts)
"java/maven"
run: https://github.com/apache/netbeans/actions/runs/30568040974/attempts/1 (2 attempts)
"java.hints batch1"
run: https://github.com/apache/netbeans/actions/runs/30016462865/attempts/1 (2 attempts)
"ide/lsp.client"
</details>
its now mostly `java.source.base
`
the concrete tests were:
- 5x `testTaggedControllerDoesNotBreakPartialReparse: org.netbeans.modules.java.source.parsing.PartialReparseTest`
- 2x `testQueryIndexRefreshQueryAgain: org.netbeans.api.java.source.ClassIndexTest`
- and 2x `org.netbeans.api.java.source.ScanUtilsTest` but in different test cases
similar observation in https://github.com/apache/netbeans/pull/9585#issuecomment-5503907628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2870480452
race condition in `webcommon/javascript2.editor` tests was fixed via https://github.com/apache/netbeans/pull/8455
unreliable `java.hints` batch 1/2, `webcommon/html.knockout` and `ide/css.editor` tests were isolated and wrapped in retry scripts via https://github.com/apache/netbeans/pull/8450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "sdedic (via GitHub)" <gi...@apache.org>.
sdedic commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2611776802
I take java/maven and will try platform/o.n.bootstrap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2614097065
extracted `java.hints batch1` failures into separate issue since this might be a real bug/be more difficult to fix -> https://github.com/apache/netbeans/issues/8195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [I] Improve test reliability [netbeans]
Posted by "mbien (via GitHub)" <gi...@apache.org>.
mbien commented on issue #8183:
URL: https://github.com/apache/netbeans/issues/8183#issuecomment-2613781060
for `groovy/groovy.editor` -> https://github.com/apache/netbeans/pull/8191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists