You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ReemaAlzaid (via GitHub)" <gi...@apache.org> on 2026/06/15 23:14:57 UTC
[I] [VL] Support multi threaded asynchronous data upload [gluten]
ReemaAlzaid opened a new issue, #12303:
URL: https://github.com/apache/gluten/issues/12303
### Description
## Description
Support multi threaded asynchronous data upload to object storage.
Currently, object storage uploads can become a bottleneck when writing large files because multipart upload parts are uploaded synchronously. This limits throughput, especially for remote object stores where each part upload has network latency.
This feature proposes adding support for asynchronous multipart uploads so multiple parts can be uploaded concurrently.
Ref: https://github.com/apache/gluten/issues/12022, https://github.com/facebookincubator/velox/pull/14472
### Configuration
```text
spark.gluten.velox.s3UploadPartAsync
spark.gluten.velox.s3MaxConcurrentUploadNum
spark.gluten.velox.s3UploadThreads
```
These would map to Velox S3 configs such as:
```text
hive.s3.part-upload-async
hive.s3.max-concurrent-upload-num
hive.s3.upload-threads
```
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support multi threaded asynchronous data upload [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid closed issue #12303: [VL] Support multi threaded asynchronous data upload
URL: https://github.com/apache/gluten/issues/12303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support multi threaded asynchronous data upload [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid commented on issue #12303:
URL: https://github.com/apache/gluten/issues/12303#issuecomment-4713247192
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org