You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "awdavidson (via GitHub)" <gi...@apache.org> on 2023/05/30 09:13:50 UTC
[GitHub] [incubator-uniffle] awdavidson opened a new pull request, #914: [#881] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
awdavidson opened a new pull request, #914:
URL: https://github.com/apache/incubator-uniffle/pull/914
What changes were proposed in this pull request?
Ensure all events are marked as understorage, this will result to the LocalStorageMeta being updated when events are processed.
Why are the changes needed?
Currently LocalStorageMeta is only update with metrics from the first event in a given shuffleId and partitionId, the first event updates metrics because there is no entry in partitionsOfStorage and the event get marked as underStorage, however, for future events in the same shuffleId and partitionId selectStorage returns the storage and does not mark the event as underStorage so when updateWriteMetrics is called, event.getUnderStorage() returns null and storage.updateWriteMetrics(metrics); is skipped.
As metrics are not updated correctly, LocalStorage.canWrite will not return the correct result.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added unit test which covers multi events for the same shuffleId and partitionId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi merged pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1578978714
@awdavidson What's your company? Do you have wechat account? We have Wechat group. If you want to join the Wechat group, you can scan the QR code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1578600389
@awdavidson Could you fix this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] awdavidson commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "awdavidson (via GitHub)" <gi...@apache.org>.
awdavidson commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1578965401
> @awdavidson Could you fix this issue?
@jerqi please can you approve the workflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] awdavidson commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "awdavidson (via GitHub)" <gi...@apache.org>.
awdavidson commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1579093952
> @awdavidson What's your company? Do you have Wechat account? We have aWechat group. If you want to join the Wechat group, you can scan the QR code.
> 
I’ve been investigating in my personal time currently not related to employer. I do not have a WeChat, I’ll consider getting one.
Thanks for merge 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1579120791
> I’ve been investigating in my personal time currently not related to employer.
If you don't have wechat, we can also use slack to communicate
https://join.slack.com/t/the-asf/shared_invite/zt-1fm9561yr-uzTpjqg3jf5nxSJV5AE3KQ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] awdavidson commented on pull request #914: [#881] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "awdavidson (via GitHub)" <gi...@apache.org>.
awdavidson commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1568077057
@jerqi @zuston as requested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #914: [#881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #914:
URL: https://github.com/apache/incubator-uniffle/pull/914#issuecomment-1579001322
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/914?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#914](https://app.codecov.io/gh/apache/incubator-uniffle/pull/914?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (e6c5b3d) into [branch-0.7](https://app.codecov.io/gh/apache/incubator-uniffle/commit/5017585032f13697396d07638d5f20d9ec71044a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (5017585) will **increase** coverage by `0.01%`.
> The diff coverage is `66.66%`.
```diff
@@ Coverage Diff @@
## branch-0.7 #914 +/- ##
================================================
+ Coverage 60.75% 60.76% +0.01%
- Complexity 1795 1796 +1
================================================
Files 214 214
Lines 12383 12385 +2
Branches 1047 1048 +1
================================================
+ Hits 7523 7526 +3
+ Misses 4455 4454 -1
Partials 405 405
```
| [Impacted Files](https://app.codecov.io/gh/apache/incubator-uniffle/pull/914?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...he/uniffle/server/storage/LocalStorageManager.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/914?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9zdG9yYWdlL0xvY2FsU3RvcmFnZU1hbmFnZXIuamF2YQ==) | `86.18% <66.66%> (+0.15%)` | :arrow_up: |
... and [2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-uniffle/pull/914/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org