You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2023/02/25 01:41:45 UTC
[GitHub] [incubator-uniffle] zuston opened a new pull request, #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
zuston opened a new pull request, #657:
URL: https://github.com/apache/incubator-uniffle/pull/657
### What changes were proposed in this pull request?
Fix memory check failure due to the execution sequence in ShuffleBufferManagerTest#bufferSizeTest
### Why are the changes needed?
Flaky test fix: #408
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#discussion_r1118219857
##########
server/src/test/java/org/apache/uniffle/server/buffer/ShuffleBufferManagerTest.java:
##########
@@ -621,5 +621,10 @@ private void waitForFlush(ShuffleFlushManager shuffleFlushManager,
fail("Flush data time out");
}
} while (committedCount < expectedBlockNum);
+
+ // Need to wait for `event.doCleanup` to be executed
+ // to ensure the correctness of subsequent checks of
+ // `shuffleBufferManager.getUsedMemory()` and `shuffleBufferManager.getInFlushSize()`.
Review Comment:
SGTM. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#issuecomment-1445922292
@zuston Could you cherry-pick and commit to branch 0.7?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#issuecomment-1444899109
# [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#657](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de28749) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/18c738ea60b32c733f7f7b23b09ad6b0d83a116d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (18c738e) will **increase** coverage by `2.29%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #657 +/- ##
============================================
+ Coverage 60.60% 62.90% +2.29%
- Complexity 1799 1801 +2
============================================
Files 216 202 -14
Lines 12458 10494 -1964
Branches 1052 1052
============================================
- Hits 7550 6601 -949
+ Misses 4500 3547 -953
+ Partials 408 346 -62
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rnetes/operator/pkg/webhook/inspector/inspector.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL2luc3BlY3Rvci5nbw==) | | |
| [...oy/kubernetes/operator/pkg/controller/util/util.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL2NvbnRyb2xsZXIvdXRpbC91dGlsLmdv) | | |
| [deploy/kubernetes/operator/pkg/utils/util.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL3V0aWwuZ28=) | | |
| [...bernetes/operator/pkg/controller/controller/rss.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL2NvbnRyb2xsZXIvY29udHJvbGxlci9yc3MuZ28=) | | |
| [...pkg/controller/sync/shuffleserver/shuffleserver.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL2NvbnRyb2xsZXIvc3luYy9zaHVmZmxlc2VydmVyL3NodWZmbGVzZXJ2ZXIuZ28=) | | |
| [...eploy/kubernetes/operator/pkg/utils/coordinator.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL2Nvb3JkaW5hdG9yLmdv) | | |
| [deploy/kubernetes/operator/pkg/utils/config.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL2NvbmZpZy5nbw==) | | |
| [deploy/kubernetes/operator/pkg/webhook/manager.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svbWFuYWdlci5nbw==) | | |
| [deploy/kubernetes/operator/pkg/utils/certs.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL2NlcnRzLmdv) | | |
| [...y/kubernetes/operator/pkg/webhook/inspector/rss.go](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL3Jzcy5nbw==) | | |
| ... and [5 more](https://codecov.io/gh/apache/incubator-uniffle/pull/657?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#discussion_r1117867437
##########
server/src/test/java/org/apache/uniffle/server/buffer/ShuffleBufferManagerTest.java:
##########
@@ -621,5 +621,10 @@ private void waitForFlush(ShuffleFlushManager shuffleFlushManager,
fail("Flush data time out");
}
} while (committedCount < expectedBlockNum);
+
+ // Need to wait for `event.doCleanup` to be executed
+ // to ensure the correctness of subsequent checks of
+ // `shuffleBufferManager.getUsedMemory()` and `shuffleBufferManager.getInFlushSize()`.
+ Thread.sleep(100);
Review Comment:
is there some way to ensure event.doCleanup is executed?
This still could be flaky. But I'm ok to merge if there's no easy way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#discussion_r1117890249
##########
server/src/test/java/org/apache/uniffle/server/buffer/ShuffleBufferManagerTest.java:
##########
@@ -621,5 +621,10 @@ private void waitForFlush(ShuffleFlushManager shuffleFlushManager,
fail("Flush data time out");
}
} while (committedCount < expectedBlockNum);
+
+ // Need to wait for `event.doCleanup` to be executed
+ // to ensure the correctness of subsequent checks of
+ // `shuffleBufferManager.getUsedMemory()` and `shuffleBufferManager.getInFlushSize()`.
Review Comment:
We use `awaitility` here better.
await().atMost(5, SECONDS).until(shuffleBufferManager.getUsedMemory() == 0);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657#discussion_r1117867437
##########
server/src/test/java/org/apache/uniffle/server/buffer/ShuffleBufferManagerTest.java:
##########
@@ -621,5 +621,10 @@ private void waitForFlush(ShuffleFlushManager shuffleFlushManager,
fail("Flush data time out");
}
} while (committedCount < expectedBlockNum);
+
+ // Need to wait for `event.doCleanup` to be executed
+ // to ensure the correctness of subsequent checks of
+ // `shuffleBufferManager.getUsedMemory()` and `shuffleBufferManager.getInFlushSize()`.
+ Thread.sleep(100);
Review Comment:
is there some to ensure event.doCleanup is executed?
This still could be flaky. But I'm ok to merge if there's no easy way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston merged pull request #657: [#408] test: fix memory check failure in ShuffleBufferManagerTest#bufferSizeTest
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #657:
URL: https://github.com/apache/incubator-uniffle/pull/657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org