You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "chasingegg (via GitHub)" <gi...@apache.org> on 2025/06/26 03:51:12 UTC
[I] [C++][s3] Does arrow support real async read interface? [arrow]
chasingegg opened a new issue, #46913:
URL: https://github.com/apache/arrow/issues/46913
### Describe the enhancement requested
Hi I want to use arrow to async-read s3 files, but seems we only have the default readAsync interface and its implementation is like a sync IO on a background thread, not a real async implementation. Do you have any plans to support it, thx!
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "chasingegg (via GitHub)" <gi...@apache.org>.
chasingegg commented on issue #46913:
URL: https://github.com/apache/arrow/issues/46913#issuecomment-3017599922
@pitrou could we relay on s3CrtClient which is more efficient instead of s3Client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46913:
URL: https://github.com/apache/arrow/issues/46913#issuecomment-3018087602
> Haha, that's true... trying to do so, hopefully there are not too much to change between two clients' interfaces.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46913:
URL: https://github.com/apache/arrow/issues/46913#issuecomment-3018026525
> [@pitrou](https://github.com/pitrou) I guess a `real async` operation should call s3's getObjectAsync to let s3 (crt)client to deal with it.
Well, AFAIU, it would use the AWS SDK's own thread pool, wouldn't it?
> [@pitrou](https://github.com/pitrou) could we rely on s3CrtClient which is more efficient instead of s3Client?
There's already an issue open for that: https://github.com/apache/arrow/issues/18923
Someone has to work on a PR for it, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "chasingegg (via GitHub)" <gi...@apache.org>.
chasingegg commented on issue #46913:
URL: https://github.com/apache/arrow/issues/46913#issuecomment-3018084957
> > [@pitrou](https://github.com/pitrou) I guess a `real async` operation should call s3's getObjectAsync to let s3 (crt)client to deal with it.
>
> Well, AFAIU, it would use the AWS SDK's own thread pool, wouldn't it?
>
From my investigation, yes for s3Client, No for s3CrtClient which is easily occupying the full bandwidth without tuning the thread pool size.
> > [@pitrou](https://github.com/pitrou) could we rely on s3CrtClient which is more efficient instead of s3Client?
>
> There's already an issue open for that: [#18923](https://github.com/apache/arrow/issues/18923)
>
> Someone has to work on a PR for it, though.
Haha, that's true... trying to do so, hopefully there are not too much to change between two clients' interfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #46913: [C++][s3] Does arrow support real async read interface?
URL: https://github.com/apache/arrow/issues/46913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][s3] Does arrow support real async read interface? [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46913:
URL: https://github.com/apache/arrow/issues/46913#issuecomment-4843147656
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org