You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by GitBox <gi...@apache.org> on 2022/12/08 18:53:06 UTC
[GitHub] [directory-scimple] aditya-kumbhar opened a new pull request, #200: Fix flaky tests in PhoneNumberBuilderTest
aditya-kumbhar opened a new pull request, #200:
URL: https://github.com/apache/directory-scimple/pull/200
### Motivation
PhoneNumberBuilderTest has the following tests that are flaky, detected using the [NonDex](https://github.com/TestingResearchIllinois/NonDex) tool:
org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest#test_adding_params_for_GlobalPhoneNumberBuilder
org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest#test_adding_params_for_LocalPhoneNumberBuilder
The tests fail due to non-deterministic ordering in the HashMap `params` of the `PhoneNumber` class.
### Command to reproduce
`mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Denforcer.skip=true -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest`
### Assertions that fail
```
assertEquals(("tel:+1-888-888-5555;ext=1234;milhouse=simpson;example=gh234"), phoneNumber.getValue());
assertEquals(("tel:888-5555;isub=example.a.com;phone-context=+1-888;milhouse=simpson;example=gh234"), phoneNumber.getValue());
```
### Log
```
[ERROR] Failures:
[ERROR] PhoneNumberBuilderTest.test_adding_params_for_GlobalPhoneNumberBuilder:712 expected: <tel:+1-888-888-5555;ext=1234;milhouse=simpson;example=gh234> but was: <tel:+1-888-888-5555;ext=1234;example=gh234;milhouse=simpson>
[ERROR] PhoneNumberBuilderTest.test_adding_params_for_LocalPhoneNumberBuilder:728 expected: <tel:888-5555;[isub=example.a.com](http://isub%3Dexample.a.com/);phone-context=+1-888;milhouse=simpson;example=gh234> but was: <tel:888-5555;[isub=example.a.com](http://isub%3Dexample.a.com/);phone-context=+1-888;example=gh234;milhouse=simpson>
```
### Changes
The non-deterministic part of `phoneNumber.getValue()` is the params, so assert that `phoneNumber.getValue()`contains the params in such a way that the order of the params is not assumed.
Another way to fix the tests is to change the datatype of `param` from HashMap to LinkedHashMap in the `PhoneNumber` class as LinkedHashMap maintains the order of its elements - but this would require changing the main code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
[GitHub] [directory-scimple] bdemers commented on pull request #200: Fix flaky tests in PhoneNumberBuilderTest
Posted by GitBox <gi...@apache.org>.
bdemers commented on PR #200:
URL: https://github.com/apache/directory-scimple/pull/200#issuecomment-1343295364
Thanks @aditya-kumbhar!
I when the LinkedHashMap route, but added a note that it could be a SortedMap in the future.
nondex is a pretty cool Maven plugin! I tried running it on the rest of the project but encountered problems.
The biggest one is it wasn't obvious how to debug the test run. Any chance you can add an example to the README/wiki of the project that shows how to run a test and attach a debugger? (or even better, run the instrumented test from an IDE like IntelliJ)
Updating the plugin to work with Java 17+ is critical too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
[GitHub] [directory-scimple] bdemers closed pull request #200: Fix flaky tests in PhoneNumberBuilderTest
Posted by GitBox <gi...@apache.org>.
bdemers closed pull request #200: Fix flaky tests in PhoneNumberBuilderTest
URL: https://github.com/apache/directory-scimple/pull/200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
[GitHub] [directory-scimple] aditya-kumbhar commented on pull request #200: Fix flaky tests in PhoneNumberBuilderTest
Posted by GitBox <gi...@apache.org>.
aditya-kumbhar commented on PR #200:
URL: https://github.com/apache/directory-scimple/pull/200#issuecomment-1345490364
Thanks @bdemers for the feedback!
Nondex will be updated to work with 17+ in a [later version](https://github.com/TestingResearchIllinois/NonDex/issues/179).
Regarding debugging/attaching a debugger while using Nondex, this method works for me (in IntelliJ):
1. Add [remote debug](https://spin.atomicobject.com/2020/08/20/maven-debugging-intellij/#:~:text=In%20the%20IDE%2C%20add%20debug,command%2C%20replacing%20mvn%20with%20mvnDebug%20.&text=In%20IntelliJ%2C%20make%20sure%20your,press%20the%20%E2%80%9CDebug%E2%80%9D%20button.) configuration in IntelliJ with port 5005.
2. Add debug breakpoints.
3. Run `mvn nondex` with ` -Dmaven.surefire.debug`. For example, to run the command in this PR in debug mode:
`mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dmaven.surefire.debug -Denforcer.skip=true -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest`
The mvn command should attach to the IntelliJ debugger.
Another note: to reproduce a flaky test detected using Nondex without having to run all the Nondex iterations again, we can use the reported Nondex seed and pass it as a parameter.
For example, if the reported Nondex seed is 933178, following command can be used to replicate the Nondex configuration and reproduce the test failure:
`
mvn -pl scim-spec/scim-spec-schema edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dmaven.surefire.debug -Dtest=org.apache.directory.scim.spec.resources.PhoneNumberBuilderTest -DnondexSeed=933178
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org