You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "maobaolong (via GitHub)" <gi...@apache.org> on 2024/09/25 14:51:44 UTC
[PR] [MINOR] improve(client): Reduce reassignResult log size within driver [incubator-uniffle]
maobaolong opened a new pull request, #2147:
URL: https://github.com/apache/incubator-uniffle/pull/2147
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[#123] feat(operator): support xxx"
- "[#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
Reduce the empty size server in the reassignResult.
### Why are the changes needed?
Reduce the log size.
Without this PR, the size of driver log is huge. There are large size log content like this.
```
{
192.168.1.100-19977-17000={818=[], 819=[], 803=[], 968=[], 857=[], 906=[], 826=[], 939=[], 956=[], 845=[]},
192.168.1.101-19977-17000={193=[], 65=[], 3=[], 68=[], 134=[], 135=[], 139=[], 76=[], 12=[], 78=[], 17=[], 82=[], 19=[], 151=[], 91=[], 29=[], 94=[], 160=[], 33=[], 167=[], 168=[], 173=[], 177=[], 178=[], 115=[], 183=[], 58=[]},
192.168.1.102-19977-17000={288=[], 294=[], 262=[], 361=[], 267=[], 332=[], 333=[], 238=[], 270=[], 207=[], 336=[], 241=[], 372=[], 278=[], 280=[], 282=[], 380=[], 221=[], 349=[], 318=[]},
192.168.1.103-19977-17000={1187=[], 1096=[], 1097=[], 1130=[], 1003=[], 1197=[], 1006=[], 1199=[], 1137=[], 1075=[], 1076=[], 1044=[], 1142=[], 1019=[], 1052=[], 1149=[]},
192.168.1.104-19977-17000={450=[], 484=[], 420=[], 581=[], 518=[], 459=[], 493=[], 463=[], 433=[], 498=[], 434=[], 530=[], 467=[], 435=[], 596=[], 470=[], 471=[], 439=[], 472=[], 408=[], 475=[], 446=[], 542=[], 511=[]},
192.168.1.105-19977-17000={1280=[], 1385=[], 1386=[], 1322=[], 1293=[], 1360=[], 1329=[], 1396=[], 1205=[], 1302=[], 1208=[], 1241=[], 1341=[], 1311=[]},
192.168.1.106-19977-17000={1505=[], 1410=[], 1444=[], 1481=[], 1577=[], 1418=[], 1580=[], 1585=[], 1426=[], 1554=[], 1427=[], 1493=[], 1432=[], 1529=[], 1434=[], 1402=[], 1535=[], 1471=[]},
192.168.1.107-19977-17000={1602=[], 1700=[], 1669=[], 1766=[], 1606=[], 1798=[], 1639=[], 1609=[], 1610=[], 1709=[], 1677=[], 1679=[], 1747=[], 1651=[], 1716=[], 1783=[], 1720=[], 1689=[], 1722=[], 1791=[], 1663=[]}, 30.118.176.221-19977-17000={705=[], 673=[], 641=[], 611=[], 773=[], 743=[], 616=[], 681=[], 649=[], 783=[], 754=[], 755=[], 787=[], 757=[], 600=[], 699=[], 635=[]},
192.168.1.108-19977-17000={1888=[], 1957=[], 1830=[], 1800=[], 1993=[], 1834=[], 1963=[], 1869=[], 1998=[], 1967=[], 1840=[], 1809=[], 1812=[], 1909=[], 1881=[], 1882=[], 1850=[], 1818=[], 1851=[], 1951=[]}}
```
After this PR, the empty `updatedReassignServers`, should never added to the `reassignResult` , so the log can be tiny.
### Does this PR introduce _any_ user-facing change?
(Please list the user-facing changes introduced by your change, including
1. Change in user-facing APIs.
2. Addition or removal of property keys.)
No.
### How was this patch tested?
(Please test your changes, and provide instructions on how to test it:
1. If you add a feature or fix a bug, add a test to cover your changes.
2. If you fix a flaky test, repeat it for many times to prove it works.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [MINOR] improve(client): Reduce reassignResult log size within driver [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2147:
URL: https://github.com/apache/incubator-uniffle/pull/2147#issuecomment-2374402577
## Test Results
2 342 files - 448 2 342 suites - 448 2h 17m 8s :stopwatch: - 3h 7m 21s
901 tests - 118 898 :white_check_mark: - 117 1 :zzz: ±0 2 :x: - 1
11 625 runs - 1 039 11 580 :white_check_mark: - 1 058 15 :zzz: ±0 30 :x: +19
For more details on these failures, see [this check](https://github.com/apache/incubator-uniffle/runs/30653784362).
Results for commit dd3a7dee. ± Comparison against base commit 8f4494b6.
<details>
<summary>This pull request <b>removes</b> 118 tests.</summary>
```
org.apache.uniffle.test.AQERepartitionTest ‑ resultCompareTest
org.apache.uniffle.test.AQESkewedJoinTest ‑ resultCompareTest
org.apache.uniffle.test.AQESkewedJoinWithLocalOrderTest ‑ resultCompareTest
org.apache.uniffle.test.AccessClusterTest ‑ testUsingCustomExtraProperties
org.apache.uniffle.test.AccessClusterTest ‑ test{File}
org.apache.uniffle.test.AssignmentWithTagsTest ‑ testTags
org.apache.uniffle.test.AutoAccessTest ‑ test
org.apache.uniffle.test.CombineByKeyTest ‑ combineByKeyTest
org.apache.uniffle.test.ContinuousSelectPartitionStrategyTest ‑ resultCompareTest
org.apache.uniffle.test.CoordinatorAdminServiceTest ‑ test
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAIAq9GYC/02Myw6DIBBFf8Ww7oIZRLA/01AGElIfDcLK9N8LiaLLe+7J2ZkPk9vYs0PR46NjWw7pvilHk8K6FKIRdSHlTlUYOZzrtWVrqzHqC33Ct6DL8SZMtduAi3GNhfBCYl5qE2BAecwzCiA1b+yoNumoimbcq3ad55DKYETCKHJu8IgAymgt3r63akTvjJJSAREnrdjvD47bKMASAQAA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [MINOR] improve(client): Reduce reassignResult log size within driver [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #2147:
URL: https://github.com/apache/incubator-uniffle/pull/2147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org