You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "shouwangyw (via GitHub)" <gi...@apache.org> on 2025/03/17 15:55:35 UTC
[PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
shouwangyw opened a new pull request, #3157:
URL: https://github.com/apache/celeborn/pull/3157
### What changes were proposed in this pull request?
- close(CELEBORN-1916)[https://issues.apache.org/jira/browse/CELEBORN-1916]
- This PR extends the Multipart Uploader (MPU) interface to support Aliyun OSS.
### Why are the changes needed?
- Implemented multipart-uploader-oss module based on the existing MPU extension interface.
- Added necessary configurations and dependencies for Aliyun OSS integration.
- Ensured compatibility with the existing multipart-uploader framework.
- This enhancement allows seamless multipart upload functionality for Aliyun OSS, similar to the existing AWS S3 support.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Deployment integration testing has been completed in the local environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2749983521
This PR LGTM overall. Add some notes to the readme.md can be helpful for users who want to use this feature. There can be some performance optimizations, but they can be achieved in other PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2760081824
Can you provide some test results for this PR? For example, 10G terasort tests could be enough. We can compare the performance with the OSS and the local disks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#discussion_r2032419968
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -1110,7 +1110,7 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
fileName,
hdfsFileInfo)
return (hdfsFlusher.get, hdfsFileInfo, null)
- } else if (dirs.isEmpty && location.getStorageInfo.S3Available()) {
Review Comment:
Should not change this behavior here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2731438167
Thanks. There are many UT failures, you'll need to fix them first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2785098178
I am testing this PR and will update it if I find anything that needs updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #3157: [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface
URL: https://github.com/apache/celeborn/pull/3157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "shouwangyw (via GitHub)" <gi...@apache.org>.
shouwangyw commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2750047379
> This PR LGTM overall. Add some notes to the readme.md can be helpful for users who want to use this feature. There can be some performance optimizations, but they can be achieved in other PRs.
Yes, I added some notes for using shuffle oss in the readme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1916] Support Aliyun OSS Based on MPU Extension Interface [celeborn]
Posted by "shouwangyw (via GitHub)" <gi...@apache.org>.
shouwangyw commented on PR #3157:
URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2764943445
> Can you provide some test results for this PR? For example, 10G terasort tests could be enough. We can compare the performance with the OSS and the local disks.
I performed the following test:
1. Spark Submission Command：
`spark-shell --conf spark.driver.cores=1 --conf spark.driver.memory=1G --conf spark.executor.instances=4 --conf spark.executor.cores=1 --conf spark.executor.memory=2G --conf spark.celeborn.storage.availableTypes=OSS`
2. Generating ~20GB of Shuffle Data:
spark.sparkContext.parallelize(1 to 200, 200).flatMap(_ => (1 to 15000000).iterator.map(num => num)).repartition(20).count
3. Test Screenshot:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org