You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2025/05/30 09:25:58 UTC
[PR] [CELEBORN-2021] Fix issues on regression HDFS and OSS before release 0.6 [celeborn]
FMX opened a new pull request, #3302:
URL: https://github.com/apache/celeborn/pull/3302
### What changes were proposed in this pull request?
1. Fix a NPE when reading HDFS files.
2. Change partition manager will generate correct storage info.
3. Add assertions for tier writers.
### Why are the changes needed?
Regression for release 0.6.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Cluster tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2021] Fix issues on regression HDFS and OSS before release 0.6 [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #3302: [CELEBORN-2021] Fix issues on regression HDFS and OSS before release 0.6
URL: https://github.com/apache/celeborn/pull/3302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2021] Fix issues on regression HDFS and OSS before release 0.6 [celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3302:
URL: https://github.com/apache/celeborn/pull/3302#issuecomment-2921817826
## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3302?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 63.63%. Comparing base [(`fff9725`)](https://app.codecov.io/gh/apache/celeborn/commit/fff97252ad7ef59ff68046e0a33392df453aded7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`4971505`)](https://app.codecov.io/gh/apache/celeborn/commit/49715054d5b7ade36471a60468eb050dbf776bd2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 48 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3302 +/- ##
==========================================
+ Coverage 63.54% 63.63% +0.10%
==========================================
Files 343 343
Lines 20812 20868 +56
Branches 1835 1842 +7
==========================================
+ Hits 13222 13278 +56
+ Misses 6630 6629 -1
- Partials 960 961 +1
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3302?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2021] Fix issues on regression HDFS and OSS before release 0.6 [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3302:
URL: https://github.com/apache/celeborn/pull/3302#discussion_r2125271883
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StoragePolicy.scala:
##########
@@ -138,7 +138,7 @@ class StoragePolicy(conf: CelebornConf, storageManager: StorageManager, source:
partitionDataWriterContext.isPartitionSplitEnabled)
partitionDataWriterContext.setWorkingDir(workingDir)
val metaHandler = getPartitionMetaHandler(diskFileInfo)
- if (storageInfoType == StorageInfo.Type.HDD || storageInfoType == StorageInfo.Type.SSD) {
+ if (storageInfoType == StorageInfo.Type.HDD || storageInfoType == StorageInfo.Type.SSD && location.getStorageInfo.localDiskAvailable()) {
Review Comment:
`((storageInfoType == StorageInfo.Type.HDD || storageInfoType == StorageInfo.Type.SSD) && location.getStorageInfo.localDiskAvailable())`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org