You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "jerqi (via GitHub)" <gi...@apache.org> on 2023/03/10 09:29:26 UTC
[GitHub] [incubator-uniffle] jerqi opened a new pull request, #705: App manager close
jerqi opened a new pull request, #705:
URL: https://github.com/apache/incubator-uniffle/pull/705
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[#123] feat(operator): support xxx"
- "[#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
(Please outline the changes and how this PR fixes the issue.)
### Why are the changes needed?
(Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, describe the bug.)
Fix: # (issue)
### Does this PR introduce _any_ user-facing change?
(Please list the user-facing changes introduced by your change, including
1. Change in user-facing APIs.
2. Addition or removal of property keys.)
No.
### How was this patch tested?
(Please test your changes, and provide instructions on how to test it:
1. If you add a feature or fix a bug, add a test to cover your changes.
2. If you fix a flaky test, repeat it for many times to prove it works.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on pull request #705: [#483] test: fix flaky test ShuffleServerFaultToleranceTest
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#issuecomment-1469282568
I still got this error in
https://github.com/kaijchen/incubator-uniffle/actions/runs/4422573893/jobs/7754489189#step:6:3546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132404732
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
There is no `bind error`. But it occurs that server is unavailable. I suspect that the port is available when the server bind, but the port is unavailable when the server send data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132391839
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
Why `+ 20` fix the problem? Can we set it to zero for any available port?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132412742
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
No, `shutdownNow` just make the log more clearer, otherwise thread which haven't shutdown will log in other test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132392973
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
Why `+ 20` fix the problem? Can we set it to zero for any available port?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132391839
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
Why `+ 20` fix the problem? Can we set it to zero for any available port?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi merged pull request #705: [#483] test: fix flaky test ShuffleServerFaultToleranceTest
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132409449
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
> There is no `bind error`. But it occurs that server is unavailable.
So why is it related to the failure, is `shutdownNow` enough to fix the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132428878
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
OK, I'm testing it by repeating CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #705: [#483] fix: Modify the ports of flaky test
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#discussion_r1132404732
##########
integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerFaultToleranceTest.java:
##########
@@ -244,7 +244,7 @@ public static MockedShuffleServer createServer(int id, File tmpDir) throws Excep
String basePath = dataDir1.getAbsolutePath() + "," + dataDir2.getAbsolutePath();
shuffleServerConf.set(ShuffleServerConf.RSS_STORAGE_TYPE, StorageType.MEMORY_LOCALFILE_HDFS.name());
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 450L);
- shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + id);
+ shuffleServerConf.setInteger("rss.rpc.server.port", SHUFFLE_SERVER_PORT + 20 + id);
Review Comment:
There is no `bind error`. But it occurs that server is unavailable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #705: [#483] test: fix flaky test ShuffleServerFaultToleranceTest
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#issuecomment-1464035754
Merged master & branch 0.7, thanks @kaijchen , I tried, but I don't find the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #705: [MINOR] fix: Modify the ports of flaky test
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #705:
URL: https://github.com/apache/incubator-uniffle/pull/705#issuecomment-1463530267
## [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/705?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#705](https://codecov.io/gh/apache/incubator-uniffle/pull/705?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6a0f1e0) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/ffa50b979ea6ae11b423646cc0c2381af29e4cf8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ffa50b9) will **increase** coverage by `1.89%`.
> The diff coverage is `66.66%`.
```diff
@@ Coverage Diff @@
## master #705 +/- ##
============================================
+ Coverage 60.80% 62.69% +1.89%
- Complexity 1840 1847 +7
============================================
Files 221 214 -7
Lines 12648 10780 -1868
Branches 1062 1067 +5
============================================
- Hits 7690 6758 -932
+ Misses 4552 3674 -878
+ Partials 406 348 -58
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/705?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../apache/uniffle/coordinator/CoordinatorServer.java](https://codecov.io/gh/apache/incubator-uniffle/pull/705?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvQ29vcmRpbmF0b3JTZXJ2ZXIuamF2YQ==) | `59.23% <50.00%> (+1.85%)` | :arrow_up: |
| [...apache/uniffle/coordinator/ApplicationManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/705?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvQXBwbGljYXRpb25NYW5hZ2VyLmphdmE=) | `83.88% <71.42%> (-0.69%)` | :arrow_down: |
... and [24 files with indirect coverage changes](https://codecov.io/gh/apache/incubator-uniffle/pull/705/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org