You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by "KangweiZhu (via GitHub)" <gi...@apache.org> on 2025/10/09 04:42:37 UTC

[PR] Fix Flaky Tests in GraphSONTypedCompatibilityTest [tinkerpop]

KangweiZhu opened a new pull request, #3237:
URL: https://github.com/apache/tinkerpop/pull/3237

   We observed several tests in GraphSONTypedCompatibilityTest  that exhibited flaky behavior when executed with NonDex. Specifically speaking, we can reproduce them by using the following commands.
   ```
   mvn clean install -DskipTests -Drat.skip=true
   ```
   * Test shouldReadWriteEdge[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteEdge[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v3)]" -Drat.skip=true
   ```
   
   * Test shouldReadWritePath[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWritePath[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v3)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteProperty[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteProperty[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v3)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteTraverser[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteTraverser[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v3)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteVertexProperty[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteVertexProperty[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v3)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteVertex[expect(v2)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v2)]" -Drat.skip=true
   ```
   
   * Test shouldReadWriteVertex[expect(v3)]
   ```bash
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v3)]" -Drat.skip=true
   ```
   And the error should be something like this:
   ```
   [ERROR] Failures:
   [ERROR]   GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]>
   [ERROR]   GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]>
   ```
   
   Upon our investigation, the root cause is the use of:
   ```
   protected Map<Object, Vertex> vertices = new ConcurrentHashMap<>();
   protected Map<Object, Edge> edges = new ConcurrentHashMap<>();
   ```
   in TinkerGraph.java, which does not guarantee a deterministic order.
   
   The simplest fix would be to replace ConcurrentHashMap with LinkedHashMap, as we did in a previous PR. We've confirmed that this change could remove the flakiness of these tests. However, we are concerned that such a change might introduce unintended side effects in the code under test. Another possible fix would be to deterministically select a fixed id, but that approach would make the test become sensitive to future implementation changes. Thus, we decided to use this sorting-based approach to deflake this test.
   
   <!--
   Thanks for contributing! Reminders:
   + TARGET the earliest branch where you want the change
       3.7-dev -> 3.7.5 (non-breaking only)
       3.8-dev -> 3.8.0
       master  -> 4.0.0
   + Committers will MERGE the PR forward to newer versions
   + ADD entry to the CHANGELOG.asciidoc for the targeted version
       Do not reference a JIRA number there
   + ADD JIRA number to title and link in description
   + PRs requires 3 +1s from committers OR
                  1 +1 and 7 day wait to merge.
   + MORE details: https://s.apache.org/rtnal
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Deflake tests in GraphSONTypedCompatibilityTest, GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest [tinkerpop]

Posted by "Cole-Greer (via GitHub)" <gi...@apache.org>.
Cole-Greer merged PR #3237:
URL: https://github.com/apache/tinkerpop/pull/3237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Deflake tests in GraphSONTypedCompatibilityTest, GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest [tinkerpop]

Posted by "KangweiZhu (via GitHub)" <gi...@apache.org>.
KangweiZhu commented on PR #3237:
URL: https://github.com/apache/tinkerpop/pull/3237#issuecomment-3439758996

   We made another change that deflakes serveral tests in GraphSONUntypedCompatibilityTest.
   
   In addition, we  found the scope of this fix is broader than we initially thought, so we updated the title of this PR and modified the commit message.
   This pr actually also fixed all flaky tests in GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest, and could be verified by:
   ```
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUntypedCompatibilityTest" -Drat.skip=true
   ```
   ```
   mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphbinary.GraphBinaryCompatibilityTest" -Drat.skip=true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Flaky Tests in GraphSONTypedCompatibilityTest [tinkerpop]

Posted by "andreachild (via GitHub)" <gi...@apache.org>.
andreachild commented on code in PR #3237:
URL: https://github.com/apache/tinkerpop/pull/3237#discussion_r2421237749


##########
gremlin-util/src/test/java/org/apache/tinkerpop/gremlin/structure/io/Model.java:
##########
@@ -115,12 +116,22 @@ private Model() {
         addCoreEntry(new java.sql.Timestamp(1481750076295L), "Timestamp", "");
         addCoreEntry(UUID.fromString("41d2e28a-20a4-4ab0-b379-d810dede3786"), "UUID");
 
-        addGraphStructureEntry(graph.edges().next(), "Edge", "");
-        addGraphStructureEntry(g.V().out().out().path().next(), "Path", "");
-        addGraphStructureEntry(graph.edges().next().properties().next(), "Property", "");
+        addGraphStructureEntry(IteratorUtils.list(graph.edges()).stream()
+                .sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id()))
+                .iterator().next(), "Edge", "");
+        addGraphStructureEntry(g.V().order().by(T.id).out().out().path().next(), "Path", "");
+        addGraphStructureEntry(IteratorUtils.list(graph.edges()).stream()
+                .sorted((e1, e2) -> Integer.compare((Integer)e1.id(), (Integer)e2.id()))
+                .iterator().next().properties().next(), "Property", "");

Review Comment:
   Should the properties be sorted as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Flaky Tests in GraphSONTypedCompatibilityTest [tinkerpop]

Posted by "Cole-Greer (via GitHub)" <gi...@apache.org>.
Cole-Greer commented on PR #3237:
URL: https://github.com/apache/tinkerpop/pull/3237#issuecomment-3407420484

   VOTE +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Flaky Tests in GraphSONTypedCompatibilityTest [tinkerpop]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3237:
URL: https://github.com/apache/tinkerpop/pull/3237#issuecomment-3386973288

   ## [Codecov](https://app.codecov.io/gh/apache/tinkerpop/pull/3237?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 74.24%. Comparing base ([`9b46b67`](https://app.codecov.io/gh/apache/tinkerpop/commit/9b46b6777d2fa250e41daacf2fa4554605aff53a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5364eee`](https://app.codecov.io/gh/apache/tinkerpop/commit/5364eeebeb74199c42a69fc972c523f9bb59d1fc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 382 commits behind head on 3.7-dev.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             3.7-dev    #3237      +/-   ##
   =============================================
   - Coverage      76.14%   74.24%   -1.91%     
   =============================================
     Files           1084       28    -1056     
     Lines          65160     5917   -59243     
     Branches        7285        0    -7285     
   =============================================
   - Hits           49616     4393   -45223     
   + Misses         12839     1322   -11517     
   + Partials        2705      202    -2503     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/tinkerpop/pull/3237?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org