You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "everpcpc (via GitHub)" <gi...@apache.org> on 2024/10/15 09:38:11 UTC
[PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
everpcpc opened a new pull request, #5184:
URL: https://github.com/apache/opendal/pull/5184
# Which issue does this PR close?
Closes #4471
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking-changes` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5184:
URL: https://github.com/apache/opendal/pull/5184#discussion_r1802678373
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
+pub trait HttpFetch: Send + Sync {
+ /// Fetch a request in async way.
+ async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>>;
+}
+
+#[derive(Clone)]
+struct ReqwestHttpFetcher {
Review Comment:
We can do this in `opendal`. Users can implement via a wrapper.
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
+pub trait HttpFetch: Send + Sync {
+ /// Fetch a request in async way.
+ async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>>;
+}
+
+#[derive(Clone)]
+struct ReqwestHttpFetcher {
Review Comment:
We can do this in `opendal`. Users can implement via a wrapper outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #5184:
URL: https://github.com/apache/opendal/pull/5184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "flaneur2020 (via GitHub)" <gi...@apache.org>.
flaneur2020 commented on code in PR #5184:
URL: https://github.com/apache/opendal/pull/5184#discussion_r1802650946
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
Review Comment:
just curious, why it's a must to have a standalone Dyn trait? 😲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5184:
URL: https://github.com/apache/opendal/pull/5184#discussion_r1802394032
##########
core/src/raw/http_util/client.rs:
##########
@@ -20,21 +20,26 @@ use std::fmt::Formatter;
use std::future;
use std::mem;
use std::str::FromStr;
+use std::sync::Arc;
use futures::TryStreamExt;
use http::Request;
use http::Response;
+use once_cell::sync::Lazy;
use raw::oio::Read;
use super::parse_content_encoding;
use super::parse_content_length;
use super::HttpBody;
use crate::*;
+/// Http client used across opendal for loading credentials.
+pub static CREDENTIAL_CLIENT: Lazy<reqwest::Client> = Lazy::new(reqwest::Client::new);
Review Comment:
How about naming it `GLOBAL_REQWEST_CLIENT`?
Please add in comments that this is merely a temporary solution because `reqsign` requires a `reqwest` client to be passed. We will remove it after the next major version of `reqsign`, which will enable users to provide their own client.
##########
core/src/raw/http_util/client.rs:
##########
@@ -47,26 +52,24 @@ impl Debug for HttpClient {
impl HttpClient {
/// Create a new http client in async context.
pub fn new() -> Result<Self> {
- Self::build(reqwest::ClientBuilder::new())
+ let fetcher = Arc::new(ReqwestHttpFetcher::new());
+ Ok(Self { fetcher })
}
/// Construct `Self` with given [`reqwest::Client`]
- pub fn with(client: reqwest::Client) -> Self {
- Self { client }
+ pub fn with(client: impl HttpFetch + 'static) -> Self {
Review Comment:
We can remove `'static`.
##########
core/src/raw/http_util/mod.rs:
##########
@@ -24,6 +24,7 @@
mod client;
pub use client::HttpClient;
+pub use client::CREDENTIAL_CLIENT;
Review Comment:
Please mark this as `pub(crate)`. I prefer not to expose it to our users, and it's highly likely that we'll remove it in the future.
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
+pub trait HttpFetch: Send + Sync {
+ /// Fetch a request in async way.
+ async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>>;
+}
+
+#[derive(Clone)]
+struct ReqwestHttpFetcher {
Review Comment:
Maybe we can implement `HttpFetch` for `reqwest::Client` directly?
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
Review Comment:
Let's implement it in the same way as we do for `oio::Read`:
https://github.com/apache/opendal/blob/9f5f3ce00812b0a07e7648fac66bc4a01c1048d9/core/src/raw/oio/read/api.rs#L44-L105
We will have a `HttpFetch` and `HttpFetchDyn`
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
+pub trait HttpFetch: Send + Sync {
Review Comment:
Please also add `Unpin + 'static` here, we don't allow users to implement `HttpFetch` over ref.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5184:
URL: https://github.com/apache/opendal/pull/5184#discussion_r1802591927
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +90,41 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+pub trait HttpFetch: Send + Sync + Unpin + 'static {
Review Comment:
Please add comments to this trait. This trait is exposed to users, and we expect that users should never interact with `HttpFetchDyn`.
##########
core/src/raw/http_util/client.rs:
##########
@@ -19,22 +19,36 @@ use std::fmt::Debug;
use std::fmt::Formatter;
use std::future;
use std::mem;
+use std::ops::Deref;
use std::str::FromStr;
+use std::sync::Arc;
+use futures::Future;
use futures::TryStreamExt;
use http::Request;
use http::Response;
+use once_cell::sync::Lazy;
use raw::oio::Read;
use super::parse_content_encoding;
use super::parse_content_length;
use super::HttpBody;
+use crate::raw::*;
use crate::*;
+/// Http client used across opendal for loading credentials.
+/// This is merely a temporary solution because reqsign requires a reqwest client to be passed.
+/// We will remove it after the next major version of reqsign, which will enable users to provide their own client.
+#[allow(dead_code)]
+pub(crate) static GLOBAL_REQWEST_CLIENT: Lazy<reqwest::Client> = Lazy::new(reqwest::Client::new);
+
+/// HttpFetcher is a type erased [`HttpFetch`].
+pub type HttpFetcher = Box<dyn HttpFetchDyn>;
Review Comment:
We can use `Arc<dyn HttpFetchDyn>` here to eliminate the need for an extra `Box`.
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +90,41 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+pub trait HttpFetch: Send + Sync + Unpin + 'static {
+ /// Fetch a request in async way.
+ fn fetch(
+ &self,
+ req: Request<Buffer>,
+ ) -> impl Future<Output = Result<Response<HttpBody>>> + MaybeSend;
+}
+
+/// HttpFetchDyn is the dyn version of [`HttpFetch`]
+/// which make it possible to use as `Box<dyn HttpFetchDyn>`
+pub trait HttpFetchDyn: Send + Sync + Unpin + 'static {
+ /// The dyn version of [`HttpFetch::fetch`].
+ ///
+ /// This function returns a boxed future to make it object safe.
+ fn fetch_dyn(&self, req: Request<Buffer>) -> BoxedFuture<Result<Response<HttpBody>>>;
+}
+
+impl<T: HttpFetch + ?Sized> HttpFetchDyn for T {
+ fn fetch_dyn(&self, req: Request<Buffer>) -> BoxedFuture<Result<Response<HttpBody>>> {
+ Box::pin(self.fetch(req))
+ }
+}
+
+impl<T: HttpFetchDyn + ?Sized> HttpFetch for Box<T> {
Review Comment:
Implement for `Arc<T>` instead.
##########
core/src/raw/http_util/mod.rs:
##########
@@ -25,6 +25,10 @@
mod client;
pub use client::HttpClient;
+/// temporary client used by several features
+#[allow(unused_imports)]
+pub(crate) use client::GLOBAL_REQWEST_CLIENT;
Review Comment:
Please make `HttpFetch` public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): abstract HttpFetch trait for raw http client [opendal]
Posted by "flaneur2020 (via GitHub)" <gi...@apache.org>.
flaneur2020 commented on code in PR #5184:
URL: https://github.com/apache/opendal/pull/5184#discussion_r1802644471
##########
core/src/raw/http_util/client.rs:
##########
@@ -78,6 +81,34 @@ impl HttpClient {
/// Fetch a request in async way.
pub async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>> {
+ self.fetcher.fetch(req).await
+ }
+}
+
+#[async_trait::async_trait]
+pub trait HttpFetch: Send + Sync {
+ /// Fetch a request in async way.
+ async fn fetch(&self, req: Request<Buffer>) -> Result<Response<HttpBody>>;
+}
+
+#[derive(Clone)]
+struct ReqwestHttpFetcher {
Review Comment:
can we implement trait for other crate's struct? 😲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org