You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "morristai (via GitHub)" <gi...@apache.org> on 2023/11/27 07:36:03 UTC
[PR] feat(core): service add HuggingFace file system [incubator-opendal]
morristai opened a new pull request, #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670
## Description
<!-- Provide a brief summary of the changes introduced by this PR. -->
close: #3605
## Addition Note
- HuggingFace doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [HuggingFace Hub's source code](https://github.com/huggingface/huggingface_hub).
- For `message.rs`, I'm unsure whether we should drop unused fields or add `#[allow(dead_code)]` to disable warnings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1412777753
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
Review Comment:
Oh, should have written this at the beginning. There's no possibility of error return here. Thank you, changed!
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
+ None => Ok("main".to_string()),
+ }?;
+ debug!("backend use revision: {}", &revision);
+
+ let root = normalize_root(&self.config.root.take().unwrap_or_default());
+ debug!("backend use root: {}", &root);
+
+ let token = self.config.token.as_ref().cloned();
+
+ let client = HttpClient::new()?;
+
+ debug!("backend build finished: {:?}", &self);
+ Ok(HuggingfaceBackend {
+ core: Arc::new(HuggingfaceCore {
+ repo_type,
+ repo_id,
+ revision,
+ root,
+ token,
+ client,
+ }),
+ })
+ }
+}
+
+/// Backend for Huggingface service
+#[derive(Debug, Clone)]
+pub struct HuggingfaceBackend {
+ core: Arc<HuggingfaceCore>,
+}
+
+#[async_trait]
+impl Accessor for HuggingfaceBackend {
+ type Reader = IncomingAsyncBody;
+ type BlockingReader = ();
+ type Writer = ();
+ type BlockingWriter = ();
+ type Lister = oio::PageLister<HuggingfaceLister>;
+ type BlockingLister = ();
+
+ fn info(&self) -> AccessorInfo {
+ let mut am = AccessorInfo::default();
+ am.set_scheme(Scheme::Huggingface)
+ .set_native_capability(Capability {
+ stat: true,
+
+ read: true,
+ read_can_next: true,
+ read_with_range: true,
+
+ list: true,
+ list_without_recursive: true,
+ list_with_recursive: true,
+
+ ..Default::default()
+ });
+ am
+ }
+
+ async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> {
+ let resp = self.core.hf_read(path, args).await?;
+
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK => {
+ let size = parse_content_length(resp.headers())?;
+ Ok((RpRead::new().with_size(size), resp.into_body()))
+ }
+ StatusCode::RANGE_NOT_SATISFIABLE => Ok((RpRead::new(), IncomingAsyncBody::empty())),
+ _ => Err(parse_error(resp).await?),
+ }
+ }
+
+ async fn stat(&self, path: &str, _: OpStat) -> Result<RpStat> {
+ // Stat root always returns a DIR.
+ if path == "/" {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ let resp = self.core.hf_path_info(path).await?;
+
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK => {
+ let mut meta = parse_into_metadata(path, resp.headers())?;
+ let bs = resp.into_body().bytes().await?;
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingfaceStatus>>(&bs)
+ .map_err(new_json_deserialize_error)?;
+
+ // NOTE: if the file is not found, the server will return 200 with an empty array
+ if let Some(status) = decoded_response.get(0) {
+ if let Some(commit_info) = status.last_commit.as_ref() {
+ meta.set_last_modified(parse_datetime_from_rfc3339(
+ commit_info.date.as_str(),
+ )?);
+ }
+
+ match status.type_.as_str() {
+ "directory" => meta.set_mode(EntryMode::DIR),
+ "file" => meta.set_mode(EntryMode::FILE),
+ _ => return Err(Error::new(ErrorKind::Unexpected, "unknown status type")),
+ };
+ } else if path.ends_with('/') {
Review Comment:
Dropped
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1833144447
> Seems not fixed. With ConfigDeserializer, we don't need to use write map.get("abc") by hand.
(It appears that I am unable to reply in that review block, so I'll respond here.)
@Xuanwo, Sorry, it's my negligence. But I want to point out that I'm referring to service `Etcd`. It seems like it's also miss refactored?
https://github.com/apache/incubator-opendal/blob/0711e6ce97660380628b72a010d092fd29ec9553/core/src/services/etcd/backend.rs#L200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1829986812
> I just wanted to make sure you know about it :)
You are so nice! And thanks for the review. :heart:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1405774415
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingFaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("read_token").map(|v| builder.read_token(v));
+
+ builder
+ }
+
+ /// Build a HuggingFaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match &self.repo_type {
Review Comment:
How about writing in this way:
```rust
match self.repo_type.as_deref() {
Some("model") => ..,
Some("dataset") => ..,
Some("space") => ..,
_ => ...
}
```
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
How about naming it `token`? Will hf have different token types? And maybe we should provide more comment for this, like what's the benefits to fill if token?
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
Review Comment:
Do we really need to parse `repo_type` to `RepoType`? How about using it in url directly?
##########
core/src/services/huggingface/mod.rs:
##########
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod backend;
+pub use backend::HuggingFaceBuilder as HuggingFace;
Review Comment:
If we want to make it name after `HuggingFace`, we should use `hugging_face` as it scheme.
Please make sure they are consistent:
- `HuggingFace`, `hugging_face`, `hugging-face`
- `Huggingface`, `huggingface`
I prefer to use the second one.
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
Review Comment:
Please use `ConfigDeserializer`, take s3 as an example.
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.read_token {
+ let auth_header_content = format!("Bearer {}", token);
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let path = percent_encode_path(&p);
+ let req_body = &json!({
+ "paths": path,
+ "expand": "True",
+ });
+
+ // NOTE: We need to transfer body to x-www-form-urlencoded format.
+ let req_body =
+ serde_urlencoded::to_string(&req_body).expect("failed to encode request body");
Review Comment:
Can we avoid introducing `serde_urlencoded`?
##########
core/src/services/huggingface/message.rs:
##########
Review Comment:
We can move all those types into `core.rs`
##########
core/src/services/huggingface/message.rs:
##########
@@ -0,0 +1,401 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HuggingFace response messages
+
+use serde::Deserialize;
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceStatus {
+ #[serde(rename = "type")]
+ pub type_: String,
+ pub oid: String,
+ pub size: u64,
+ pub lfs: Option<HuggingFaceLfs>,
+ pub path: String,
+ pub last_commit: Option<HuggingFaceLastCommit>,
+ pub security: Option<HuggingFaceSecurity>,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceLfs {
+ pub oid: String,
+ pub size: u64,
+ pub pointer_size: u64,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceLastCommit {
+ pub id: String,
+ pub title: String,
+ pub date: String,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceSecurity {
+ pub blob_id: String,
+ pub name: String,
+ pub safe: bool,
+ pub av_scan: Option<HuggingFaceAvScan>,
+ pub pickle_import_scan: Option<HuggingFacePickleImportScan>,
+}
+
+#[derive(Deserialize)]
+#[allow(dead_code)]
+#[serde(rename_all = "camelCase")]
+pub(super) struct HuggingFaceAvScan {
+ pub virus_found: bool,
+ pub virus_names: Option<Vec<String>>,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFacePickleImportScan {
+ pub highest_safety_level: String,
+ pub imports: Vec<HuggingFaceImport>,
+}
+
+#[derive(Deserialize)]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceImport {
+ pub module: String,
+ pub name: String,
+ pub safety: String,
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::raw::new_json_deserialize_error;
+ use crate::types::Result;
+ use bytes::Bytes;
+
+ #[test]
+ fn parse_list_response_test() -> Result<()> {
+ let resp = Bytes::from(
+ r#"
+ [
+ {
+ "type": "file",
+ "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "size": 69512435,
+ "lfs": {
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "pointerSize": 133
+ },
+ "path": "maelstrom/lib/maelstrom.jar"
+ },
+ {
+ "type": "directory",
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "path": "maelstrom/lib/plugins"
+ }
+ ]
+ "#,
+ );
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingFaceStatus>>(&resp)
+ .map_err(new_json_deserialize_error)?;
+
+ assert_eq!(decoded_response.len(), 2);
+
+ assert_eq!(decoded_response[0].type_, "file");
+ assert_eq!(
+ decoded_response[0].oid,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(decoded_response[0].size, 69512435);
+ assert_eq!(decoded_response[0].path, "maelstrom/lib/maelstrom.jar");
+
+ assert_eq!(
+ decoded_response[0].lfs.as_ref().unwrap().oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().size, 69512435);
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().pointer_size, 133);
+
+ assert_eq!(decoded_response[1].type_, "directory");
+ assert_eq!(
+ decoded_response[1].oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+ assert_eq!(decoded_response[1].size, 69512435);
+ assert_eq!(decoded_response[1].path, "maelstrom/lib/plugins");
+
+ Ok(())
+ }
+
+ #[test]
+ fn parse_files_info_test() -> Result<()> {
+ let resp = Bytes::from(
+ r#"
+ [
+ {
+ "type": "file",
+ "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "size": 69512435,
+ "lfs": {
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "pointerSize": 133
+ },
+ "path": "maelstrom/lib/maelstrom.jar",
+ "lastCommit": {
+ "id": "bc1ef030bf3743290d5e190695ab94582e51ae2f",
+ "title": "Upload 141 files",
+ "date": "2023-11-17T23:50:28.000Z"
+ },
+ "security": {
+ "blobId": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "name": "maelstrom/lib/maelstrom.jar",
+ "safe": true,
+ "avScan": {
+ "virusFound": false,
+ "virusNames": null
+ },
+ "pickleImportScan": {
+ "highestSafetyLevel": "innocuous",
+ "imports": [
+ {"module": "torch", "name": "FloatStorage", "safety": "innocuous"},
+ {"module": "collections", "name": "OrderedDict", "safety": "innocuous"},
+ {"module": "torch", "name": "LongStorage", "safety": "innocuous"},
+ {"module": "torch._utils", "name": "_rebuild_tensor_v2", "safety": "innocuous"}
+ ]
+ }
+ }
+ }
+ ]
+ "#,
+ );
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingFaceStatus>>(&resp)
+ .map_err(new_json_deserialize_error)?;
+
+ assert_eq!(decoded_response.len(), 1);
+
+ assert_eq!(decoded_response[0].type_, "file");
+ assert_eq!(
+ decoded_response[0].oid,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(decoded_response[0].size, 69512435);
+ assert_eq!(decoded_response[0].path, "maelstrom/lib/maelstrom.jar");
+
+ assert_eq!(
+ decoded_response[0].lfs.as_ref().unwrap().oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().size, 69512435);
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().pointer_size, 133);
+
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().id,
+ "bc1ef030bf3743290d5e190695ab94582e51ae2f"
+ );
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().title,
+ "Upload 141 files"
+ );
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().date,
+ "2023-11-17T23:50:28.000Z"
+ );
+
+ assert_eq!(
+ decoded_response[0].security.as_ref().unwrap().blob_id,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(
+ decoded_response[0].security.as_ref().unwrap().name,
+ "maelstrom/lib/maelstrom.jar"
+ );
+ assert_eq!(decoded_response[0].security.as_ref().unwrap().safe, true);
+
+ assert_eq!(
Review Comment:
We can implement `Eq` for those types, so we can `assert_eq!(decoded_response[0], TypeAbc)` directly.
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.read_token {
+ let auth_header_content = format!("Bearer {}", token);
Review Comment:
Please use `format_authorization_by_bearer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Wauplin (via GitHub)" <gi...@apache.org>.
Wauplin commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407843341
##########
core/src/services/huggingface/docs.md:
##########
@@ -0,0 +1,63 @@
+This service will visit the [HuggingFace API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to access the HuggingFace File System.
+Currently, we only support the `model` and `dataset` types of repositories, and operations are limited to reading and listing/stating.
+
+HuggingFace doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [HuggingFace Hub](https://github.com/huggingface/huggingface_hub).
Review Comment:
(for context) Technically, `huggingface_hub` is the name of the Python library to interact with the Hugging Face Hub (the online platform). So https://huggingface.co/docs/huggingface_hub/package_reference/hf_api is the package reference of the Python library (which is a good starting point for retro engineering). There is some documentation about the API itself (see [here](https://huggingface.co/docs/hub/api)) but it's not complete yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Wauplin (via GitHub)" <gi...@apache.org>.
Wauplin commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407828605
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
I would also name it `token` to be consistent with the existing huggingface ecosystem (not only the HF libraries but also third-party libraries integrated with the Hub). At the moment, tokens are mostly `read` or `write` but it is not set in stone. We recently introduced in beta tokens that have other types of permissions (`api-inference` for instance). So I wouldn't differentiate token types here and name everything `token`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1412778069
##########
core/src/services/huggingface/lister.rs:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceLister {
+ core: Arc<HuggingfaceCore>,
+ path: String,
+ recursive: bool,
+}
+
+impl HuggingfaceLister {
+ pub fn new(core: Arc<HuggingfaceCore>, path: String, recursive: bool) -> Self {
+ Self {
+ core,
+ path,
+ recursive,
+ }
+ }
+}
+
+#[async_trait]
+impl oio::PageList for HuggingfaceLister {
+ async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> {
+ let response = self.core.hf_list(&self.path, self.recursive).await?;
+
+ let status_code = response.status();
+ if !status_code.is_success() {
+ let error = parse_error(response).await?;
+ return Err(error);
+ }
+
+ let bytes = response.into_body().bytes().await?;
+ let decoded_response = serde_json::from_slice::<Vec<HuggingfaceStatus>>(&bytes)
+ .map_err(new_json_deserialize_error)?;
+
+ ctx.done = true;
+
+ for status in decoded_response {
+ let entry: oio::Entry = match status.type_.as_str() {
+ "directory" => {
+ let normalized_path = format!("{}/", &status.path);
+ let mut meta = Metadata::new(EntryMode::DIR);
+ if let Some(commit_info) = status.last_commit.as_ref() {
Review Comment:
Thank you, refactored
##########
core/src/types/scheme.rs:
##########
@@ -303,6 +307,7 @@ impl FromStr for Scheme {
"gridfs" => Ok(Scheme::Gridfs),
"hdfs" => Ok(Scheme::Hdfs),
"http" | "https" => Ok(Scheme::Http),
+ "huggingface" => Ok(Scheme::Huggingface),
Review Comment:
Done
##########
core/src/types/scheme.rs:
##########
@@ -357,6 +362,7 @@ impl From<Scheme> for &'static str {
Scheme::Gridfs => "gridfs",
Scheme::Hdfs => "hdfs",
Scheme::Http => "http",
+ Scheme::Huggingface => "hf",
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Wauplin (via GitHub)" <gi...@apache.org>.
Wauplin commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407828605
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
I would also name it `token` to be consistent with the existing huggingface ecosystem (not only the HF libraries but also third-party libraries integrated with the Hub). At the moment, tokens are mostly `read` or `write` but it is not set in stone. We recently introduced tokens that have other types of permissions (`api-inference` for instance). So I wouldn't differentiate token types here and name everything `token`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1412778005
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,415 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde::Deserialize;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingfaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingfaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingfaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let req_body = format!("paths={}&expand=True", percent_encode_path(&p));
+
+ let req = req
+ .body(AsyncBody::Bytes(Bytes::from(req_body)))
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_list(
+ &self,
+ path: &str,
+ recursive: bool,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let mut url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ if recursive {
+ url.push_str("&recursive=True");
+ }
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let req = req
+ .body(AsyncBody::Empty)
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_read(&self, path: &str, arg: OpRead) -> Result<Response<IncomingAsyncBody>> {
Review Comment:
Renamed `hf_read` to `hf_resolve`
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,415 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde::Deserialize;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingfaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingfaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingfaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let req_body = format!("paths={}&expand=True", percent_encode_path(&p));
+
+ let req = req
+ .body(AsyncBody::Bytes(Bytes::from(req_body)))
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_list(
+ &self,
+ path: &str,
+ recursive: bool,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let mut url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ if recursive {
+ url.push_str("&recursive=True");
+ }
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let req = req
+ .body(AsyncBody::Empty)
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_read(&self, path: &str, arg: OpRead) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/{}/resolve/{}/{}",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/datasets/{}/resolve/{}/{}",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let range = arg.range();
+ if !range.is_full() {
+ req = req.header("Range", &range.to_header());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1835760001
> Yes, there is an on-going effort to refactor them one by one: https://github.com/apache/incubator-opendal/issues/3240
I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that `Etcd` has already been refactored, but the `from_map` part seems to have been overlooked. Could you please verify this from your end? I'm not certain if I've missed something here. (By the way, I am currently on a business trip, so my response may be delayed.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1829966859
> I also wanted to point you to https://github.com/huggingface/hf-hub which is the rust client to interact with the Hub. Though it's not as complete as the Python equivalent, it already has support for most file read/write use cases -including compatibility with the [HF cache system](https://huggingface.co/docs/huggingface_hub/guides/manage-cache)- .
Thank you for the advice, `hf-hub` is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Wauplin (via GitHub)" <gi...@apache.org>.
Wauplin commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1829982241
> Thank you for the advice, hf-hub is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.
Fair enough @Xuanwo! I just wanted to make sure you know about it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407870314
##########
core/src/types/scheme.rs:
##########
@@ -357,6 +362,7 @@ impl From<Scheme> for &'static str {
Scheme::Gridfs => "gridfs",
Scheme::Hdfs => "hdfs",
Scheme::Http => "http",
+ Scheme::HuggingFace => "huggingface",
Review Comment:
LGTM! We can register `hd` as an alternative of `huggingface`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410109511
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.read_token {
+ let auth_header_content = format!("Bearer {}", token);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410128930
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
Review Comment:
I don't have a "common sense" for the usage either. :laughing:
I can only tell you about my feelings: It looks like we can use the user input directly in the urls with simple if-else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1833180517
> Sorry, it's my negligence. But I want to point out that I'm referring to service `Etcd`. It seems like it's also miss refactored?
Yes, there is an on-going effort to refactor them one by one: https://github.com/apache/incubator-opendal/issues/3240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1835822434
> I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that `Etcd` has already been refactored, but the `from_map` part seems to have been overlooked. Could you please verify this from your end?
Seems etcd missed this part, we need to fix it.
> By the way, I am currently on a business trip, so my response may be delayed
Enjoy your trip!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407027309
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
I feel like we don't need to separate the read and write token. We don't support write operations now and we don't have a plan to do so. We can add them in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410128930
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
Review Comment:
I don't have a "common sense" for the usage either. :laughing:
I can only tell you about my feelings: It looks like we can use the user input directly in the urls with simple if-else.
Maybe we can maintain a mapping like hf_hub does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410209773
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
Review Comment:
> Clippy complains about the absence of specified generic arguments.
We don't need to return `Ok(abc)` here. How about in this way:
```rust
let revision = match &self.config.revision {
Some(revision) => revision.clone(),
None => "main".to_string,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410204421
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
Review Comment:
Clippy complains about the absence of specified generic arguments.
```shell
error[E0282]: type annotations needed
--> core/src/services/huggingface/backend.rs:214:31
|
214 | Some(revision) => Ok(revision.clone()),
| ^^ cannot infer type of the type parameter `E` declared on the enum `Result`
|
help: consider specifying the generic arguments
|
214 | Some(revision) => Ok::<std::string::String, E>(revision.clone()),
| ++++++++++++++++++++++++++
error[E0283]: type annotations needed
--> core/src/services/huggingface/backend.rs:214:31
|
214 | Some(revision) => Ok(revision.clone()),
| ^^ cannot infer type of the type parameter `E` declared on the enum `Result`
215 | None => Ok("main".to_string()),
216 | }?;
| - type must be known at this point
|
note: multiple `impl`s satisfying `types::error::Error: std::convert::From<_>` found
--> core/src/services/etcd/backend.rs:397:1
|
397 | impl From<EtcdError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
::: core/src/services/ftp/err.rs:24:1
|
24 | impl From<FtpError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
::: core/src/services/redis/backend.rs:449:1
|
449 | impl From<RedisError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
::: core/src/services/rocksdb/backend.rs:194:1
|
194 | impl From<rocksdb::Error> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
::: core/src/services/sftp/error.rs:31:1
|
31 | impl From<SftpClientError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
49 | impl From<SshError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
67 | impl From<SftpError> for Error {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: and more `impl`s found in the following crates: `core`:
- impl<T> std::convert::From<!> for T;
- impl<T> std::convert::From<T> for T;
= note: required for `std::result::Result<services::huggingface::backend::HuggingfaceBackend, types::error::Error>` to implement `std::ops::FromResidual<std::result::Result<std::convert::Infallible, _>>`
help: consider specifying the generic arguments
|
214 | Some(revision) => Ok::<std::string::String, E>(revision.clone()),
| ++++++++++++++++++++++++++
```
I'll look into this problem tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1837118382
I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata
Would you like to add a behaivor test for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410109392
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingFaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("read_token").map(|v| builder.read_token(v));
+
+ builder
+ }
+
+ /// Build a HuggingFaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match &self.repo_type {
Review Comment:
Done, I will be more careful next time. 😂
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingFaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("read_token").map(|v| builder.read_token(v));
+
+ builder
+ }
+
+ /// Build a HuggingFaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match &self.repo_type {
Review Comment:
Done, I will be more careful next time. 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410108340
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
Renamed `read_token` with `token`.
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410130889
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
Review Comment:
Why we need this?
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use super::lister::HuggingfaceLister;
+use crate::raw::*;
+use crate::*;
+
+/// Configuration for Huggingface service support.
+#[derive(Default, Deserialize, Clone)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct HuggingfaceConfig {
+ /// Repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ pub repo_type: Option<String>,
+ /// Repo id of this backend.
+ ///
+ /// This is required.
+ pub repo_id: Option<String>,
+ /// Revision of this backend.
+ ///
+ /// Default is main.
+ pub revision: Option<String>,
+ /// Root of this backend. Can be "/path/to/dir".
+ ///
+ /// Default is "/".
+ pub root: Option<String>,
+ /// Token of this backend.
+ ///
+ /// This is optional.
+ pub token: Option<String>,
+}
+
+impl Debug for HuggingfaceConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("HuggingfaceConfig");
+
+ if let Some(repo_type) = &self.repo_type {
+ ds.field("repo_type", &repo_type);
+ }
+ if let Some(repo_id) = &self.repo_id {
+ ds.field("repo_id", &repo_id);
+ }
+ if let Some(revision) = &self.revision {
+ ds.field("revision", &revision);
+ }
+ if let Some(root) = &self.root {
+ ds.field("root", &root);
+ }
+ if self.token.is_some() {
+ ds.field("token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+/// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingfaceBuilder {
+ config: HuggingfaceConfig,
+}
+
+impl Debug for HuggingfaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("config", &self.config);
+ ds.finish()
+ }
+}
+
+impl HuggingfaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.config.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.config.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.config.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.config.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the token of this backend.
+ ///
+ /// This is optional.
+ pub fn token(&mut self, token: &str) -> &mut Self {
+ if !token.is_empty() {
+ self.config.token = Some(token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingfaceBuilder {
+ const SCHEME: Scheme = Scheme::Huggingface;
+ type Accessor = HuggingfaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = HuggingfaceBuilder::default();
+
+ map.get("repo_type").map(|v| builder.repo_type(v));
+ map.get("repo_id").map(|v| builder.repo_id(v));
+ map.get("revision").map(|v| builder.revision(v));
+ map.get("root").map(|v| builder.root(v));
+ map.get("token").map(|v| builder.token(v));
+
+ builder
+ }
+
+ /// Build a HuggingfaceBackend.
+ fn build(&mut self) -> Result<Self::Accessor> {
+ debug!("backend build started: {:?}", &self);
+
+ let repo_type = match self.config.repo_type.as_deref() {
+ Some("model") => Ok(RepoType::Model),
+ Some("dataset") => Ok(RepoType::Dataset),
+ Some("space") => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ "repo type \"space\" is unsupported",
+ )),
+ Some(repo_type) => Err(Error::new(
+ ErrorKind::ConfigInvalid,
+ format!("unknown repo_type: {}", repo_type).as_str(),
+ )
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ None => Ok(RepoType::Model),
+ }?;
+ debug!("backend use repo_type: {:?}", &repo_type);
+
+ let repo_id = match &self.config.repo_id {
+ Some(repo_id) => Ok(repo_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Huggingface)),
+ }?;
+ debug!("backend use repo_id: {}", &repo_id);
+
+ let revision = match &self.config.revision {
+ Some(revision) => Ok::<String, Error>(revision.clone()),
+ None => Ok("main".to_string()),
+ }?;
+ debug!("backend use revision: {}", &revision);
+
+ let root = normalize_root(&self.config.root.take().unwrap_or_default());
+ debug!("backend use root: {}", &root);
+
+ let token = self.config.token.as_ref().cloned();
+
+ let client = HttpClient::new()?;
+
+ debug!("backend build finished: {:?}", &self);
+ Ok(HuggingfaceBackend {
+ core: Arc::new(HuggingfaceCore {
+ repo_type,
+ repo_id,
+ revision,
+ root,
+ token,
+ client,
+ }),
+ })
+ }
+}
+
+/// Backend for Huggingface service
+#[derive(Debug, Clone)]
+pub struct HuggingfaceBackend {
+ core: Arc<HuggingfaceCore>,
+}
+
+#[async_trait]
+impl Accessor for HuggingfaceBackend {
+ type Reader = IncomingAsyncBody;
+ type BlockingReader = ();
+ type Writer = ();
+ type BlockingWriter = ();
+ type Lister = oio::PageLister<HuggingfaceLister>;
+ type BlockingLister = ();
+
+ fn info(&self) -> AccessorInfo {
+ let mut am = AccessorInfo::default();
+ am.set_scheme(Scheme::Huggingface)
+ .set_native_capability(Capability {
+ stat: true,
+
+ read: true,
+ read_can_next: true,
+ read_with_range: true,
+
+ list: true,
+ list_without_recursive: true,
+ list_with_recursive: true,
+
+ ..Default::default()
+ });
+ am
+ }
+
+ async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> {
+ let resp = self.core.hf_read(path, args).await?;
+
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK => {
+ let size = parse_content_length(resp.headers())?;
+ Ok((RpRead::new().with_size(size), resp.into_body()))
+ }
+ StatusCode::RANGE_NOT_SATISFIABLE => Ok((RpRead::new(), IncomingAsyncBody::empty())),
+ _ => Err(parse_error(resp).await?),
+ }
+ }
+
+ async fn stat(&self, path: &str, _: OpStat) -> Result<RpStat> {
+ // Stat root always returns a DIR.
+ if path == "/" {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ let resp = self.core.hf_path_info(path).await?;
+
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK => {
+ let mut meta = parse_into_metadata(path, resp.headers())?;
+ let bs = resp.into_body().bytes().await?;
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingfaceStatus>>(&bs)
+ .map_err(new_json_deserialize_error)?;
+
+ // NOTE: if the file is not found, the server will return 200 with an empty array
+ if let Some(status) = decoded_response.get(0) {
+ if let Some(commit_info) = status.last_commit.as_ref() {
+ meta.set_last_modified(parse_datetime_from_rfc3339(
+ commit_info.date.as_str(),
+ )?);
+ }
+
+ match status.type_.as_str() {
+ "directory" => meta.set_mode(EntryMode::DIR),
+ "file" => meta.set_mode(EntryMode::FILE),
+ _ => return Err(Error::new(ErrorKind::Unexpected, "unknown status type")),
+ };
+ } else if path.ends_with('/') {
Review Comment:
We don't rely on this check anymore.
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
+ pub fn read_token(&mut self, read_token: &str) -> &mut Self {
+ if !read_token.is_empty() {
+ self.read_token = Some(read_token.to_string());
+ }
+ self
+ }
+}
+
+impl Builder for HuggingFaceBuilder {
+ const SCHEME: Scheme = Scheme::HuggingFace;
+ type Accessor = HuggingFaceBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
Review Comment:
Seems not fixed. With `ConfigDeserializer`, we don't need to use write `map.get("abc")` by hand.
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,415 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde::Deserialize;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingfaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingfaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingfaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let req_body = format!("paths={}&expand=True", percent_encode_path(&p));
+
+ let req = req
+ .body(AsyncBody::Bytes(Bytes::from(req_body)))
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_list(
+ &self,
+ path: &str,
+ recursive: bool,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let mut url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ if recursive {
+ url.push_str("&recursive=True");
+ }
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let req = req
+ .body(AsyncBody::Empty)
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_read(&self, path: &str, arg: OpRead) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/{}/resolve/{}/{}",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/datasets/{}/resolve/{}/{}",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let range = arg.range();
+ if !range.is_full() {
+ req = req.header("Range", &range.to_header());
Review Comment:
Please always use `http::header` for well-known header names.
##########
core/src/services/huggingface/lister.rs:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use super::core::HuggingfaceCore;
+use super::core::HuggingfaceStatus;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceLister {
+ core: Arc<HuggingfaceCore>,
+ path: String,
+ recursive: bool,
+}
+
+impl HuggingfaceLister {
+ pub fn new(core: Arc<HuggingfaceCore>, path: String, recursive: bool) -> Self {
+ Self {
+ core,
+ path,
+ recursive,
+ }
+ }
+}
+
+#[async_trait]
+impl oio::PageList for HuggingfaceLister {
+ async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> {
+ let response = self.core.hf_list(&self.path, self.recursive).await?;
+
+ let status_code = response.status();
+ if !status_code.is_success() {
+ let error = parse_error(response).await?;
+ return Err(error);
+ }
+
+ let bytes = response.into_body().bytes().await?;
+ let decoded_response = serde_json::from_slice::<Vec<HuggingfaceStatus>>(&bytes)
+ .map_err(new_json_deserialize_error)?;
+
+ ctx.done = true;
+
+ for status in decoded_response {
+ let entry: oio::Entry = match status.type_.as_str() {
+ "directory" => {
+ let normalized_path = format!("{}/", &status.path);
+ let mut meta = Metadata::new(EntryMode::DIR);
+ if let Some(commit_info) = status.last_commit.as_ref() {
Review Comment:
Seems we are repeating the same code three times?
##########
core/src/types/scheme.rs:
##########
@@ -303,6 +307,7 @@ impl FromStr for Scheme {
"gridfs" => Ok(Scheme::Gridfs),
"hdfs" => Ok(Scheme::Hdfs),
"http" | "https" => Ok(Scheme::Http),
+ "huggingface" => Ok(Scheme::Huggingface),
Review Comment:
"huggingface" | "hf"
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,415 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde::Deserialize;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingfaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingfaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingfaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingfaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let req_body = format!("paths={}&expand=True", percent_encode_path(&p));
+
+ let req = req
+ .body(AsyncBody::Bytes(Bytes::from(req_body)))
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_list(
+ &self,
+ path: &str,
+ recursive: bool,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let mut url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/tree/{}/{}?expand=True",
+ &self.repo_id,
+ &self.revision,
+ percent_encode_path(&p)
+ ),
+ };
+
+ if recursive {
+ url.push_str("&recursive=True");
+ }
+
+ let mut req = Request::get(&url);
+
+ if let Some(token) = &self.token {
+ let auth_header_content = format_authorization_by_bearer(token)?;
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ let req = req
+ .body(AsyncBody::Empty)
+ .map_err(new_request_build_error)?;
+
+ self.client.send(req).await
+ }
+
+ pub async fn hf_read(&self, path: &str, arg: OpRead) -> Result<Response<IncomingAsyncBody>> {
Review Comment:
The naming doesn't make sense. How about renaming to `hf_resolve`?
##########
core/src/types/scheme.rs:
##########
@@ -357,6 +362,7 @@ impl From<Scheme> for &'static str {
Scheme::Gridfs => "gridfs",
Scheme::Hdfs => "hdfs",
Scheme::Http => "http",
+ Scheme::Huggingface => "hf",
Review Comment:
Please use `huggingface`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410110210
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ RepoType::Dataset => format!(
+ "https://huggingface.co/api/datasets/{}/paths-info/{}",
+ &self.repo_id, &self.revision
+ ),
+ };
+
+ let mut req = Request::post(&url);
+
+ if let Some(token) = &self.read_token {
+ let auth_header_content = format!("Bearer {}", token);
+ req = req.header(header::AUTHORIZATION, auth_header_content);
+ }
+
+ req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded");
+
+ let path = percent_encode_path(&p);
+ let req_body = &json!({
+ "paths": path,
+ "expand": "True",
+ });
+
+ // NOTE: We need to transfer body to x-www-form-urlencoded format.
+ let req_body =
+ serde_urlencoded::to_string(&req_body).expect("failed to encode request body");
Review Comment:
Dropped `serde_urlencoded` and `json!`. Right now, I found that `format!("paths={}&expand=True", percent_encode_path(&p));` will do the job just fine.
##########
core/src/services/huggingface/message.rs:
##########
Review Comment:
Done
##########
core/src/services/huggingface/message.rs:
##########
@@ -0,0 +1,401 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HuggingFace response messages
+
+use serde::Deserialize;
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceStatus {
+ #[serde(rename = "type")]
+ pub type_: String,
+ pub oid: String,
+ pub size: u64,
+ pub lfs: Option<HuggingFaceLfs>,
+ pub path: String,
+ pub last_commit: Option<HuggingFaceLastCommit>,
+ pub security: Option<HuggingFaceSecurity>,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceLfs {
+ pub oid: String,
+ pub size: u64,
+ pub pointer_size: u64,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceLastCommit {
+ pub id: String,
+ pub title: String,
+ pub date: String,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceSecurity {
+ pub blob_id: String,
+ pub name: String,
+ pub safe: bool,
+ pub av_scan: Option<HuggingFaceAvScan>,
+ pub pickle_import_scan: Option<HuggingFacePickleImportScan>,
+}
+
+#[derive(Deserialize)]
+#[allow(dead_code)]
+#[serde(rename_all = "camelCase")]
+pub(super) struct HuggingFaceAvScan {
+ pub virus_found: bool,
+ pub virus_names: Option<Vec<String>>,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[allow(dead_code)]
+pub(super) struct HuggingFacePickleImportScan {
+ pub highest_safety_level: String,
+ pub imports: Vec<HuggingFaceImport>,
+}
+
+#[derive(Deserialize)]
+#[allow(dead_code)]
+pub(super) struct HuggingFaceImport {
+ pub module: String,
+ pub name: String,
+ pub safety: String,
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::raw::new_json_deserialize_error;
+ use crate::types::Result;
+ use bytes::Bytes;
+
+ #[test]
+ fn parse_list_response_test() -> Result<()> {
+ let resp = Bytes::from(
+ r#"
+ [
+ {
+ "type": "file",
+ "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "size": 69512435,
+ "lfs": {
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "pointerSize": 133
+ },
+ "path": "maelstrom/lib/maelstrom.jar"
+ },
+ {
+ "type": "directory",
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "path": "maelstrom/lib/plugins"
+ }
+ ]
+ "#,
+ );
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingFaceStatus>>(&resp)
+ .map_err(new_json_deserialize_error)?;
+
+ assert_eq!(decoded_response.len(), 2);
+
+ assert_eq!(decoded_response[0].type_, "file");
+ assert_eq!(
+ decoded_response[0].oid,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(decoded_response[0].size, 69512435);
+ assert_eq!(decoded_response[0].path, "maelstrom/lib/maelstrom.jar");
+
+ assert_eq!(
+ decoded_response[0].lfs.as_ref().unwrap().oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().size, 69512435);
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().pointer_size, 133);
+
+ assert_eq!(decoded_response[1].type_, "directory");
+ assert_eq!(
+ decoded_response[1].oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+ assert_eq!(decoded_response[1].size, 69512435);
+ assert_eq!(decoded_response[1].path, "maelstrom/lib/plugins");
+
+ Ok(())
+ }
+
+ #[test]
+ fn parse_files_info_test() -> Result<()> {
+ let resp = Bytes::from(
+ r#"
+ [
+ {
+ "type": "file",
+ "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "size": 69512435,
+ "lfs": {
+ "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c",
+ "size": 69512435,
+ "pointerSize": 133
+ },
+ "path": "maelstrom/lib/maelstrom.jar",
+ "lastCommit": {
+ "id": "bc1ef030bf3743290d5e190695ab94582e51ae2f",
+ "title": "Upload 141 files",
+ "date": "2023-11-17T23:50:28.000Z"
+ },
+ "security": {
+ "blobId": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2",
+ "name": "maelstrom/lib/maelstrom.jar",
+ "safe": true,
+ "avScan": {
+ "virusFound": false,
+ "virusNames": null
+ },
+ "pickleImportScan": {
+ "highestSafetyLevel": "innocuous",
+ "imports": [
+ {"module": "torch", "name": "FloatStorage", "safety": "innocuous"},
+ {"module": "collections", "name": "OrderedDict", "safety": "innocuous"},
+ {"module": "torch", "name": "LongStorage", "safety": "innocuous"},
+ {"module": "torch._utils", "name": "_rebuild_tensor_v2", "safety": "innocuous"}
+ ]
+ }
+ }
+ }
+ ]
+ "#,
+ );
+
+ let decoded_response = serde_json::from_slice::<Vec<HuggingFaceStatus>>(&resp)
+ .map_err(new_json_deserialize_error)?;
+
+ assert_eq!(decoded_response.len(), 1);
+
+ assert_eq!(decoded_response[0].type_, "file");
+ assert_eq!(
+ decoded_response[0].oid,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(decoded_response[0].size, 69512435);
+ assert_eq!(decoded_response[0].path, "maelstrom/lib/maelstrom.jar");
+
+ assert_eq!(
+ decoded_response[0].lfs.as_ref().unwrap().oid,
+ "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c"
+ );
+
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().size, 69512435);
+ assert_eq!(decoded_response[0].lfs.as_ref().unwrap().pointer_size, 133);
+
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().id,
+ "bc1ef030bf3743290d5e190695ab94582e51ae2f"
+ );
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().title,
+ "Upload 141 files"
+ );
+ assert_eq!(
+ decoded_response[0].last_commit.as_ref().unwrap().date,
+ "2023-11-17T23:50:28.000Z"
+ );
+
+ assert_eq!(
+ decoded_response[0].security.as_ref().unwrap().blob_id,
+ "45fa7c3d85ee7dd4139adbc056da25ae136a65f2"
+ );
+ assert_eq!(
+ decoded_response[0].security.as_ref().unwrap().name,
+ "maelstrom/lib/maelstrom.jar"
+ );
+ assert_eq!(decoded_response[0].security.as_ref().unwrap().safe, true);
+
+ assert_eq!(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407022804
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
> There are write tokens and read tokens. https://huggingface.co/docs/hub/security-tokens#how-to-manage-user-access-tokens However, the write token only allows to write in the repo with write permissions (owner's repo or repo of owner's organization).
Yes, as @Inokinoki mentioned, HuggingFace only has either read or write access bound with a token. I initially added both, but since we only support read access, I dropped the write_token. I'll add a comment for a clearer explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Wauplin (via GitHub)" <gi...@apache.org>.
Wauplin commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1407837175
##########
core/src/types/scheme.rs:
##########
@@ -357,6 +362,7 @@ impl From<Scheme> for &'static str {
Scheme::Gridfs => "gridfs",
Scheme::Hdfs => "hdfs",
Scheme::Http => "http",
+ Scheme::HuggingFace => "huggingface",
Review Comment:
The Python library already includes a filesystem interface (based on fsspec) that uses `hf://` as a scheme. It would be nice to align on this to be consistent between implementations. See https://huggingface.co/docs/huggingface_hub/guides/hf_file_system for more details.
##########
core/src/services/huggingface/docs.md:
##########
@@ -0,0 +1,63 @@
+This service will visit the [HuggingFace API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to access the HuggingFace File System.
+Currently, we only support the `model` and `dataset` types of repositories, and operations are limited to reading and listing/stating.
+
+HuggingFace doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [HuggingFace Hub](https://github.com/huggingface/huggingface_hub).
Review Comment:
Technically, `huggingface_hub` is the name of the Python library to interact with the Hugging Face Hub (the online platform). So https://huggingface.co/docs/huggingface_hub/package_reference/hf_api is the package reference of the Python library (which is a good starting point for retro engineering). There is some documentation about the API itself (see [here](https://huggingface.co/docs/hub/api)) but it's not complete yet.
##########
core/src/services/huggingface/mod.rs:
##########
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod backend;
+pub use backend::HuggingFaceBuilder as HuggingFace;
Review Comment:
I would also advocate for `huggingface` if possible :pray:
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
I would also name it `token` to be consistent with the existing huggingface ecosystem (not only the HF libraries but also third-party libraries integrated with the Hub). At the moment, tokens are mostly `read` or `write` it is not set in stone. We recently introduced in beta tokens that have other types of permissions (`api-inference` for instance). So I wouldn't differentiate token types here and name everything `token`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410110434
##########
core/src/services/huggingface/mod.rs:
##########
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod backend;
+pub use backend::HuggingFaceBuilder as HuggingFace;
Review Comment:
Renamed to `Huggingface`, `huggingface`.
##########
core/src/types/scheme.rs:
##########
@@ -357,6 +362,7 @@ impl From<Scheme> for &'static str {
Scheme::Gridfs => "gridfs",
Scheme::Hdfs => "hdfs",
Scheme::Http => "http",
+ Scheme::HuggingFace => "huggingface",
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1410112113
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
Review Comment:
I don't have a "common sense" of when we should use an enum and when we should not. Can you enlighten me on why it's better not to use an enum in this case? 🙏
##########
core/src/services/huggingface/core.rs:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::fmt::Debug;
+
+use bytes::Bytes;
+use http::Request;
+use http::Response;
+use http::{header, StatusCode};
+use serde_json::json;
+
+use super::backend::RepoType;
+use super::error::parse_error;
+use crate::raw::*;
+use crate::*;
+
+pub struct HuggingFaceCore {
+ pub repo_type: RepoType,
+ pub repo_id: String,
+ pub revision: String,
+ pub root: String,
+ pub read_token: Option<String>,
+
+ pub client: HttpClient,
+}
+
+impl Debug for HuggingFaceCore {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("HuggingFaceCore")
+ .field("repo_type", &self.repo_type)
+ .field("repo_id", &self.repo_id)
+ .field("revision", &self.revision)
+ .field("root", &self.root)
+ .finish_non_exhaustive()
+ }
+}
+
+impl HuggingFaceCore {
+ pub async fn hf_path_info(&self, path: &str) -> Result<Response<IncomingAsyncBody>> {
+ let p = build_abs_path(&self.root, path)
+ .trim_end_matches('/')
+ .to_string();
+
+ let url = match self.repo_type {
+ RepoType::Model => format!(
+ "https://huggingface.co/api/models/{}/paths-info/{}",
Review Comment:
I don't have a "common sense" of when we should use an enum and when we should not. Can you enlighten me on why it's better not to use an enum in this case? 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "Inokinoki (via GitHub)" <gi...@apache.org>.
Inokinoki commented on code in PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#discussion_r1406872903
##########
core/src/services/huggingface/backend.rs:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::fmt::Formatter;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+
+use super::core::HuggingFaceCore;
+use super::error::parse_error;
+use super::lister::HuggingFaceLister;
+use super::message::HuggingFaceStatus;
+use crate::raw::*;
+use crate::*;
+
+/// [HuggingFace](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support.
+#[doc = include_str!("docs.md")]
+#[derive(Default, Clone)]
+pub struct HuggingFaceBuilder {
+ repo_type: Option<String>,
+ repo_id: Option<String>,
+ revision: Option<String>,
+ root: Option<String>,
+ read_token: Option<String>,
+}
+
+impl Debug for HuggingFaceBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut ds = f.debug_struct("Builder");
+
+ ds.field("repo_type", &self.repo_type);
+ ds.field("repo_id", &self.repo_id);
+ ds.field("revision", &self.revision);
+ ds.field("root", &self.root);
+
+ if self.read_token.is_some() {
+ ds.field("read_token", &"<redacted>");
+ }
+
+ ds.finish()
+ }
+}
+
+impl HuggingFaceBuilder {
+ /// Set repo type of this backend. Default is model.
+ ///
+ /// Available values:
+ /// - model
+ /// - dataset
+ ///
+ /// Currently, only models and datasets are supported.
+ /// [Reference](https://huggingface.co/docs/hub/repositories)
+ pub fn repo_type(&mut self, repo_type: &str) -> &mut Self {
+ if !repo_type.is_empty() {
+ self.repo_type = Some(repo_type.to_string());
+ }
+ self
+ }
+
+ /// Set repo id of this backend. This is required.
+ ///
+ /// Repo id consists of the account name and the repository name.
+ ///
+ /// For example, model's repo id looks like:
+ /// - meta-llama/Llama-2-7b
+ ///
+ /// Dataset's repo id looks like:
+ /// - databricks/databricks-dolly-15k
+ pub fn repo_id(&mut self, repo_id: &str) -> &mut Self {
+ if !repo_id.is_empty() {
+ self.repo_id = Some(repo_id.to_string());
+ }
+ self
+ }
+
+ /// Set revision of this backend. Default is main.
+ ///
+ /// Revision can be a branch name or a commit hash.
+ ///
+ /// For example, revision can be:
+ /// - main
+ /// - 1d0c4eb
+ pub fn revision(&mut self, revision: &str) -> &mut Self {
+ if !revision.is_empty() {
+ self.revision = Some(revision.to_string());
+ }
+ self
+ }
+
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ if !root.is_empty() {
+ self.root = Some(root.to_string());
+ }
+ self
+ }
+
+ /// Set the read token of this backend.
+ ///
+ /// This is optional.
Review Comment:
There are write tokens and read tokens.
https://huggingface.co/docs/hub/security-tokens#how-to-manage-user-access-tokens
However, the write token only allows to write in the repo with write permissions (owner's repo or repo of owner's organization).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): service add HuggingFace file system [incubator-opendal]
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on PR #3670:
URL: https://github.com/apache/incubator-opendal/pull/3670#issuecomment-1837149486
> I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata
>
> Would you like to add a behaivor test for this?
Yes, I'd love to work on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org