You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/07/12 03:01:08 UTC
[GitHub] [incubator-uniffle] colinmjj opened a new pull request, #51: Remove experimental feature with ShuffleUploader
colinmjj opened a new pull request, #51:
URL: https://github.com/apache/incubator-uniffle/pull/51
### What changes were proposed in this pull request?
ShuffleUploader is an experimental feature which is target to merge and upload data from local disk to remote storage if local disk hasn't enough space. It is replaced by `MEMORY_LOCAL_HDFS` and should be removed to avoid maintenance in the future.
### Why are the changes needed?
The experimental feature is redundancy and we have to maintain it during update with new features about storage.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
with exist UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #51: Remove experimental feature with ShuffleUploader
Posted by GitBox <gi...@apache.org>.
jerqi commented on PR #51:
URL: https://github.com/apache/incubator-uniffle/pull/51#issuecomment-1181361174
MultiStorageFaultToleranceTest have a useful case `hdfsFallbackTest`. We shouldn't remove it. It verify that we can write data to local disk, when we fail to write data on hdfs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #51: Remove experimental feature with ShuffleUploader
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #51:
URL: https://github.com/apache/incubator-uniffle/pull/51#issuecomment-1181267158
# [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#51](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (342d372) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/cbe39c141a3d201906798033abbd33fc1c37008f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cbe39c1) will **decrease** coverage by `3.96%`.
> The diff coverage is `66.66%`.
> :exclamation: Current head 342d372 differs from pull request most recent head cdd253b. Consider uploading reports for the commit cdd253b to get more accurate results
```diff
@@ Coverage Diff @@
## master #51 +/- ##
============================================
- Coverage 56.86% 52.89% -3.97%
+ Complexity 1208 1020 -188
============================================
Files 152 140 -12
Lines 8437 7552 -885
Branches 819 732 -87
============================================
- Hits 4798 3995 -803
+ Misses 3378 3329 -49
+ Partials 261 228 -33
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...va/org/apache/uniffle/client/util/ClientUtils.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NsaWVudC91dGlsL0NsaWVudFV0aWxzLmphdmE=) | `18.18% <0.00%> (+0.79%)` | :arrow_up: |
| [...a/org/apache/uniffle/server/ShuffleServerConf.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlU2VydmVyQ29uZi5qYXZh) | `99.13% <ø> (-0.18%)` | :arrow_down: |
| [...uniffle/storage/factory/ShuffleHandlerFactory.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2ZhY3RvcnkvU2h1ZmZsZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <ø> (ø)` | |
| [...ache/uniffle/storage/util/ShuffleStorageUtils.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL3V0aWwvU2h1ZmZsZVN0b3JhZ2VVdGlscy5qYXZh) | `61.00% <ø> (-6.33%)` | :arrow_down: |
| [...a/org/apache/uniffle/storage/util/StorageType.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL3V0aWwvU3RvcmFnZVR5cGUuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...he/uniffle/server/storage/MultiStorageManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9zdG9yYWdlL011bHRpU3RvcmFnZU1hbmFnZXIuamF2YQ==) | `37.50% <100.00%> (+9.63%)` | :arrow_up: |
| [.../uniffle/server/storage/StorageManagerFactory.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9zdG9yYWdlL1N0b3JhZ2VNYW5hZ2VyRmFjdG9yeS5qYXZh) | `58.33% <100.00%> (+4.48%)` | :arrow_up: |
| [...ge/handler/impl/UploadedHdfsClientReadHandler.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2hhbmRsZXIvaW1wbC9VcGxvYWRlZEhkZnNDbGllbnRSZWFkSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
| [...er/impl/UploadedStorageHdfsShuffleReadHandler.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2hhbmRsZXIvaW1wbC9VcGxvYWRlZFN0b3JhZ2VIZGZzU2h1ZmZsZVJlYWRIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-81.58%)` | :arrow_down: |
| [...iffle/storage/handler/impl/ShuffleIndexHeader.java](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2hhbmRsZXIvaW1wbC9TaHVmZmxlSW5kZXhIZWFkZXIuamF2YQ==) | `0.00% <0.00%> (-77.56%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/incubator-uniffle/pull/51/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cbe39c1...cdd253b](https://codecov.io/gh/apache/incubator-uniffle/pull/51?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] colinmjj commented on pull request #51: Remove experimental feature with ShuffleUploader
Posted by GitBox <gi...@apache.org>.
colinmjj commented on PR #51:
URL: https://github.com/apache/incubator-uniffle/pull/51#issuecomment-1181383397
> MultiStorageFaultToleranceTest have a useful case `hdfsFallbackTest`. We shouldn't remove it. It verify that we can write data to local disk, when we fail to write data on hdfs.
ok, it should be recovered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi merged pull request #51: [improvement] Remove experimental feature with ShuffleUploader
Posted by GitBox <gi...@apache.org>.
jerqi merged PR #51:
URL: https://github.com/apache/incubator-uniffle/pull/51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org