You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "bokket (via GitHub)" <gi...@apache.org> on 2024/01/15 03:35:19 UTC
[PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
bokket opened a new pull request, #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980
part #2627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1457027808
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,337 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::{Client, IcloudPathQuery};
+use super::core::{parse_error, IcloudSigner, SessionData};
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// enable the china origin
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+ d.field("root", &self.root);
+ d.field("is_china_mainland", &self.is_china_mainland);
+ d.finish_non_exhaustive()
+ }
+}
+
+/// [IcloudDrive](https://www.icloud.com/iclouddrive/) service support.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct IcloudBuilder {
+ /// icloud config for web session request
+ pub config: IcloudConfig,
+ /// Specify the http client that used by this service.
+ ///
+ /// # Notes
+ ///
+ /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed
+ /// during minor updates.
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for IcloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+
+ d.field("config", &self.config);
+ d.finish_non_exhaustive()
+ }
+}
+
+impl IcloudBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.config.root = if root.is_empty() {
+ None
+ } else {
+ Some(root.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id
+ ///
+ /// It is required. your Apple login email, e.g. `example@gmail.com`
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.config.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id password
+ ///
+ /// It is required. your icloud login password, e.g. `password`
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.config.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ /// Trust token and ds_web_auth_token is used for temporary access to the icloudDrive API.
+ ///
+ /// Authenticate using session token
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
+ self.config.trust_token = if trust_token.is_empty() {
+ None
+ } else {
+ Some(trust_token.to_string())
+ };
+
+ self
+ }
+ /// ds_web_auth_token must be set in Session
+ ///
+ /// Avoid Two Factor Authentication
+ pub fn ds_web_auth_token(&mut self, ds_web_auth_token: &str) -> &mut Self {
+ self.config.ds_web_auth_token = if ds_web_auth_token.is_empty() {
+ None
+ } else {
+ Some(ds_web_auth_token.to_string())
+ };
+
+ self
+ }
+ /// Enable the china origin
+ /// For China, use "https://www.icloud.com.cn"
+ /// For Other region, use "https://www.icloud.com"
+ pub fn is_china_mainland(&mut self, is_china_mainland: bool) -> &mut Self {
+ self.config.is_china_mainland = is_china_mainland;
+ self
+ }
+
+ /// Specify the http client that used by this service.
+ ///
+ /// # Notes
+ ///
+ /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed
+ /// during minor updates.
+ pub fn http_client(&mut self, client: HttpClient) -> &mut Self {
+ self.http_client = Some(client);
+ self
+ }
+}
+
+impl Builder for IcloudBuilder {
+ const SCHEME: Scheme = Scheme::Icloud;
+ type Accessor = IcloudBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let config = IcloudConfig::deserialize(ConfigDeserializer::new(map))
+ .expect("config deserialize must succeed");
+ IcloudBuilder {
+ config,
+ http_client: None,
+ }
+ }
+
+ fn build(&mut self) -> Result<Self::Accessor> {
+ let root = normalize_root(&self.config.root.take().unwrap_or_default());
+
+ let apple_id = match &self.config.apple_id {
+ Some(apple_id) => Ok(apple_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "apple_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let password = match &self.config.password {
+ Some(password) => Ok(password.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let ds_web_auth_token = match &self.config.ds_web_auth_token {
+ Some(ds_web_auth_token) => Ok(ds_web_auth_token.clone()),
+ None => Err(
+ Error::new(ErrorKind::ConfigInvalid, "ds_web_auth_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud),
+ ),
+ }?;
+
+ let trust_token = match &self.config.trust_token {
+ Some(trust_token) => Ok(trust_token.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "trust_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ debug!(
+ "Icloud backend is_china_mainland {}",
+ &self.config.is_china_mainland
+ );
+
+ let client = if let Some(client) = self.http_client.take() {
+ client
+ } else {
+ HttpClient::new().map_err(|err| {
+ err.with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)
+ })?
+ };
+
+ let session_data = SessionData::new();
+
+ let core = IcloudSigner {
+ client: client.clone(),
+ data: session_data,
+ apple_id,
+ password,
+ trust_token: Some(trust_token),
+ ds_web_auth_token: Some(ds_web_auth_token),
+ is_china_mainland: self.config.is_china_mainland,
+ };
+
+ let core = Arc::new(Mutex::new(core));
+ Ok(IcloudBackend {
+ client: Arc::new(Client {
+ core: core.clone(),
+ root,
+ path_cache: PathCacher::new(IcloudPathQuery::new(client, core.clone())).with_lock(),
Review Comment:
done
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
Review Comment:
done
##########
core/src/services/icloud/client.rs:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456220524
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+// A node within the icloud Drive filesystem.
+#[derive(Clone)]
+pub enum DriveNode {
+ Folder(Folder),
+}
+
+impl DriveNode {
+ fn new_root(value: &IcloudRoot) -> Result<DriveNode> {
+ Ok(DriveNode::Folder(Folder {
+ id: Some(value.drivewsid.to_string()),
+ name: value.name.to_string(),
+ date_created: Some(value.date_created.clone()),
+ items: value.items.clone(),
+ mime_type: "Folder".to_string(),
+ }))
+ }
+
+ pub fn items(&self) -> Option<Vec<IcloudItem>> {
+ match self {
+ DriveNode::Folder(folder) => Some(folder.items.clone()),
+ }
+ }
+}
+
+impl std::fmt::Display for DriveNode {
Review Comment:
We don't need Display.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1895628875
> > `icloud` will only list the directory, so we need to let the `path_cache` support to return `parent_id` instead of not only returning `self_id`.
>
> I'm confused. To find the parent of `a/b/c` or `a/b/d/`, can't we simply look up the ID of `a/b/`?
Ok. I don’t know why `/` is an extra layer of uuid directory below, puts me into a misunderstanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1457027554
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
Review Comment:
done
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
+}
+
+impl IcloudPathQuery {
+ pub fn new(client: HttpClient, core: Arc<Mutex<IcloudSigner>>) -> Self {
+ IcloudPathQuery { client, core }
+ }
+}
+
+#[async_trait]
+impl PathQuery for IcloudPathQuery {
+ async fn root(&self) -> Result<String> {
+ Ok("FOLDER::com.apple.CloudDocs::root".to_string())
+ }
+
+ // Retrieves the root directory within the icloud Drive.
+ async fn query(&self, parent_id: &str, name: &str) -> Result<Option<String>> {
+ let mut core = self.core.lock().await;
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .clone()
+ .url;
+
+ let uri = format!("{}/retrieveItemDetailsInFolders", drive_url);
+
+ let body = json!([
+ {
+ "drivewsid": parent_id,
+ "partialData": false
+ }
+ ])
+ .to_string();
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = core.sign(Method::POST, uri, async_body).await?;
+
+ if response.status() == StatusCode::OK {
+ let body = &response.into_body().bytes().await?;
+
+ let root: Vec<IcloudRoot> =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ let node = &root[0];
+
+ let id = match node.items.iter().find(|it| it.name == name) {
+ Some(it) => Ok(Some(it.drivewsid.clone())),
+ None => Ok(None),
+ }?;
+ Ok(id)
+ } else {
+ Err(parse_error(response).await?)
+ }
+ }
+
+ async fn create_dir(&self, parent_id: &str, name: &str) -> Result<String> {
+ let mut core = self.core.lock().await;
+ let client_id = core.get_client_id();
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .url
+ .clone();
+ // https://p219-drivews.icloud.com.cn:443
Review Comment:
done
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1893927438
> Hi, path cache has been added in main, maybe you want to refactor by it instead?
Let me briefly review path_cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456252395
##########
core/src/services/icloud/client.rs:
##########
Review Comment:
That's fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1457027413
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple icloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ // You can get more information from [apple.com](https://support.apple.com/en-us/111754)
+ if self.is_china_mainland {
+ request = request.header("Origin", "https://www.icloud.com.cn");
+ request = request.header("Referer", "https://www.icloud.com.cn/");
+ } else {
+ request = request.header("Origin", "https://www.icloud.com");
+ request = request.header("Referer", "https://www.icloud.com/");
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) =
+ parse_header_to_str(response.headers(), ACCOUNT_COUNTRY_HEADER)?
+ {
+ self.data.account_country = Some(account_country.to_string());
+ }
+
+ if let Some(session_id) =
+ parse_header_to_str(response.headers(), SESSION_ID_HEADER)?
+ {
+ self.data.session_id = Some(session_id.to_string());
+ }
+ if let Some(session_token) =
+ parse_header_to_str(response.headers(), SESSION_TOKEN_HEADER)?
+ {
+ self.data.session_token = Some(session_token.to_string());
+ }
+
+ if let Some(scnt) = parse_header_to_str(response.headers(), SCNT_HEADER)? {
+ self.data.scnt = Some(scnt.to_string());
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ if let Some(cookie) = value.to_str().unwrap().split(';').next() {
+ if let Some((key, value)) = cookie.split_once('=') {
+ self.data
+ .cookies
+ .insert(String::from(key), String::from(value));
+ }
+ }
+ }
+ }
+ match response.status() {
+ StatusCode::UNAUTHORIZED => Err(parse_error(response).await?),
+ _ => Ok(response),
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized request",
+ )),
+ }
+ }
+}
+
+pub async fn parse_error(resp: Response<IncomingAsyncBody>) -> Result<Error> {
+ let (parts, body) = resp.into_parts();
+ let bs = body.bytes().await?;
+
+ let mut kind = match parts.status.as_u16() {
+ //status:421 Misdirected Request
+ 421 | 450 | 500 => ErrorKind::NotFound,
+ 401 => ErrorKind::Unexpected,
+ _ => ErrorKind::Unexpected,
+ };
+
+ let (message, icloud_err) = serde_json::from_reader::<_, IcloudError>(bs.clone().reader())
+ .map(|icloud_err| (format!("{icloud_err:?}"), Some(icloud_err)))
+ .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None));
+
+ if let Some(icloud_err) = &icloud_err {
+ kind = match icloud_err.status_code.as_str() {
+ "NOT_FOUND" => ErrorKind::NotFound,
+ "PERMISSION_DENIED" => ErrorKind::PermissionDenied,
+ _ => ErrorKind::Unexpected,
+ }
+ }
+
+ let mut err = Error::new(kind, &message);
+
+ err = with_error_response_context(err, parts);
+
+ Ok(err)
+}
+
+#[derive(Default, Debug, Deserialize)]
+#[allow(dead_code)]
+struct IcloudError {
+ status_code: String,
+ message: String,
+}
+
+#[derive(Default, Deserialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudWebservicesResponse {
+ #[serde(default)]
+ pub hsa_challenge_required: bool,
+ #[serde(default)]
+ pub hsa_trusted_browser: bool,
+ pub webservices: Webservices,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Webservices {
+ pub drivews: Drivews,
+ pub docws: Docws,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Drivews {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Docws {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<IcloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudObject {
+ #[serde(rename = "document_id")]
+ pub document_id: String,
+ #[serde(rename = "item_id")]
+ pub item_id: String,
+ #[serde(rename = "owner_dsid")]
+ pub owner_dsid: i64,
+ #[serde(rename = "data_token")]
+ pub data_token: DataToken,
+ #[serde(rename = "double_etag")]
+ pub double_etag: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct DataToken {
+ pub url: String,
+ pub token: String,
+ pub signature: String,
+ #[serde(rename = "wrapping_key")]
+ pub wrapping_key: String,
+ #[serde(rename = "reference_signature")]
Review Comment:
done
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple icloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ // You can get more information from [apple.com](https://support.apple.com/en-us/111754)
+ if self.is_china_mainland {
+ request = request.header("Origin", "https://www.icloud.com.cn");
+ request = request.header("Referer", "https://www.icloud.com.cn/");
+ } else {
+ request = request.header("Origin", "https://www.icloud.com");
+ request = request.header("Referer", "https://www.icloud.com/");
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) =
+ parse_header_to_str(response.headers(), ACCOUNT_COUNTRY_HEADER)?
+ {
+ self.data.account_country = Some(account_country.to_string());
+ }
+
+ if let Some(session_id) =
+ parse_header_to_str(response.headers(), SESSION_ID_HEADER)?
+ {
+ self.data.session_id = Some(session_id.to_string());
+ }
+ if let Some(session_token) =
+ parse_header_to_str(response.headers(), SESSION_TOKEN_HEADER)?
+ {
+ self.data.session_token = Some(session_token.to_string());
+ }
+
+ if let Some(scnt) = parse_header_to_str(response.headers(), SCNT_HEADER)? {
+ self.data.scnt = Some(scnt.to_string());
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ if let Some(cookie) = value.to_str().unwrap().split(';').next() {
+ if let Some((key, value)) = cookie.split_once('=') {
+ self.data
+ .cookies
+ .insert(String::from(key), String::from(value));
+ }
+ }
+ }
+ }
+ match response.status() {
+ StatusCode::UNAUTHORIZED => Err(parse_error(response).await?),
+ _ => Ok(response),
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized request",
+ )),
+ }
+ }
+}
+
+pub async fn parse_error(resp: Response<IncomingAsyncBody>) -> Result<Error> {
+ let (parts, body) = resp.into_parts();
+ let bs = body.bytes().await?;
+
+ let mut kind = match parts.status.as_u16() {
+ //status:421 Misdirected Request
+ 421 | 450 | 500 => ErrorKind::NotFound,
+ 401 => ErrorKind::Unexpected,
+ _ => ErrorKind::Unexpected,
+ };
+
+ let (message, icloud_err) = serde_json::from_reader::<_, IcloudError>(bs.clone().reader())
+ .map(|icloud_err| (format!("{icloud_err:?}"), Some(icloud_err)))
+ .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None));
+
+ if let Some(icloud_err) = &icloud_err {
+ kind = match icloud_err.status_code.as_str() {
+ "NOT_FOUND" => ErrorKind::NotFound,
+ "PERMISSION_DENIED" => ErrorKind::PermissionDenied,
+ _ => ErrorKind::Unexpected,
+ }
+ }
+
+ let mut err = Error::new(kind, &message);
+
+ err = with_error_response_context(err, parts);
+
+ Ok(err)
+}
+
+#[derive(Default, Debug, Deserialize)]
+#[allow(dead_code)]
+struct IcloudError {
+ status_code: String,
+ message: String,
+}
+
+#[derive(Default, Deserialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudWebservicesResponse {
+ #[serde(default)]
+ pub hsa_challenge_required: bool,
+ #[serde(default)]
+ pub hsa_trusted_browser: bool,
+ pub webservices: Webservices,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Webservices {
+ pub drivews: Drivews,
+ pub docws: Docws,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Drivews {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Docws {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<IcloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudObject {
+ #[serde(rename = "document_id")]
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1452189633
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
Review Comment:
Ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453307231
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1452259065
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub(crate) url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4().to_string()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct Session {
+ pub(crate) data: SessionData,
+ pub(crate) client: HttpClient,
+ pub(crate) apple_id: String,
+ pub(crate) password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+}
+
+impl Debug for Session {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud Session");
+ de.field("apple_id", &self.apple_id);
+ de.field("password", &self.password);
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.finish()
+ }
+}
+
+impl Session {
Review Comment:
Please use Signer to make it easier to understand for other maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456239565
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
+}
+
+impl IcloudPathQuery {
+ pub fn new(client: HttpClient, core: Arc<Mutex<IcloudSigner>>) -> Self {
+ IcloudPathQuery { client, core }
+ }
+}
+
+#[async_trait]
+impl PathQuery for IcloudPathQuery {
+ async fn root(&self) -> Result<String> {
+ Ok("FOLDER::com.apple.CloudDocs::root".to_string())
+ }
+
+ // Retrieves the root directory within the icloud Drive.
+ async fn query(&self, parent_id: &str, name: &str) -> Result<Option<String>> {
+ let mut core = self.core.lock().await;
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .clone()
+ .url;
+
+ let uri = format!("{}/retrieveItemDetailsInFolders", drive_url);
Review Comment:
But I didn't find a way to list files url separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/opendal/pull/3980#issuecomment-1899690886
> Hello @bokket, could you address the conflicts?
I solved manually, can we ignore the `Edge Test S3`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1452193034
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub(crate) url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4().to_string()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct Session {
+ pub(crate) data: SessionData,
+ pub(crate) client: HttpClient,
+ pub(crate) apple_id: String,
+ pub(crate) password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+}
+
+impl Debug for Session {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud Session");
+ de.field("apple_id", &self.apple_id);
+ de.field("password", &self.password);
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.finish()
+ }
+}
+
+impl Session {
Review Comment:
I actually prefer `Session`.
```
pub struct IcloudCore {
pub root: String,
pub client: HttpClient,
pub signer: Arc<IcloudSigner>,
}
```
or
```
pub struct IcloudCore {
pub root: String,
pub client: HttpClient,
pub session: Arc<IcloudSession>,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453144641
##########
.typos.toml:
##########
@@ -23,10 +23,13 @@
"hellow" = "hellow"
# Showed up in examples.
"thw" = "thw"
+# Showed up in test.
+"hsa" = "hsa"
[files]
extend-exclude = [
# Generated SSH Keys.
"fixtures/sftp/test_ssh_key",
"fixtures/sftp/test_ssh_key.pub",
+ "core/src/services/icloud/core.rs"
Review Comment:
Please don't do this. Ignore `hsa` is enough.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("trust_token", &self.trust_token);
Review Comment:
Please don't print them in debug.
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,648 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, with_error_response_context, AsyncBody, HttpClient,
+ IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud signer");
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.field("region", &self.region);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ #[warn(dead_code)]
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple iCloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple iCloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ if let Some(region) = &self.region {
+ request = request.header("Origin", region);
+ request = request.header("Referer", (region.to_string() + "/").as_str());
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) = response.headers().get(ACCOUNT_COUNTRY_HEADER) {
+ self.data.account_country =
+ Some(String::from(account_country.to_str().unwrap()));
+ }
+ if let Some(session_id) = response.headers().get(SESSION_ID_HEADER) {
+ self.data.session_id = Some(String::from(session_id.to_str().unwrap()));
+ }
+ if let Some(session_token) = response.headers().get(SESSION_TOKEN_HEADER) {
+ self.data.session_token = Some(String::from(session_token.to_str().unwrap()));
+ }
+
+ if let Some(scnt) = response.headers().get(SCNT_HEADER) {
+ self.data.scnt = Some(String::from(scnt.to_str().unwrap()));
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ #[warn(clippy::single_char_pattern)]
Review Comment:
Please remove this warning and chaning `";"` into `';'` instead.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
Review Comment:
No `iCloud` please, thanks.
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,648 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, with_error_response_context, AsyncBody, HttpClient,
+ IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud signer");
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.field("region", &self.region);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ #[warn(dead_code)]
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple iCloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple iCloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ if let Some(region) = &self.region {
+ request = request.header("Origin", region);
+ request = request.header("Referer", (region.to_string() + "/").as_str());
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) = response.headers().get(ACCOUNT_COUNTRY_HEADER) {
+ self.data.account_country =
+ Some(String::from(account_country.to_str().unwrap()));
+ }
+ if let Some(session_id) = response.headers().get(SESSION_ID_HEADER) {
+ self.data.session_id = Some(String::from(session_id.to_str().unwrap()));
+ }
+ if let Some(session_token) = response.headers().get(SESSION_TOKEN_HEADER) {
Review Comment:
Please use [`opendal::raw::parse_header_to_str`](https://opendal.apache.org/docs/rust/opendal/raw/fn.parse_header_to_str.html)
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+ d.field("region", &self.region);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+/// [IcloudDrive](https://www.icloud.com/iclouddrive/) service support.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct IcloudBuilder {
+ pub config: IcloudConfig,
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for IcloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+
+ d.field("config", &self.config);
+ d.finish_non_exhaustive()
+ }
+}
+
+impl IcloudBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.config.root = Some(root.to_string());
+ self
+ }
+
+ /// Your Apple id
+ ///
+ /// It is required. your Apple login email, e.g. `example@gmail.com`
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.config.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id password
+ ///
+ /// It is required. your icloud login password, e.g. `password`
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.config.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ /// Trust token and ds_web_auth_token is used for temporary access to the IcloudDrive API.
+ /// Authenticate using session token
+ /// You can get more information from [pyicloud](https://github.com/picklepete/pyicloud/tree/master?tab=readme-ov-file#authentication)
+ /// or [iCloud-API](https://github.com/MauriceConrad/iCloud-API?tab=readme-ov-file#getting-started)
+
Review Comment:
remove this line.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
Review Comment:
The term 'region' could be misleading in this context as it may be confused with the AWS S3 region setting. How about add a new config called `is_china_mainland`?
See:
- https://github.com/picklepete/pyicloud/pull/418
- https://support.apple.com/en-us/111754
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use chrono::Utc;
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{
+ parse_error, IcloudCreateFolder, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner,
+};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<chrono::DateTime<Utc>>,
+ pub date_modified: Option<chrono::DateTime<Utc>>,
+ pub mime_type: String,
+}
+
+// A directory in iCloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<chrono::DateTime<Utc>>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+impl Folder {
+ #[warn(dead_code)]
Review Comment:
This is not allowed. We can remove `Folder` struct if we don't need them for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453307498
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use chrono::Utc;
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{
+ parse_error, IcloudCreateFolder, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner,
+};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<chrono::DateTime<Utc>>,
+ pub date_modified: Option<chrono::DateTime<Utc>>,
+ pub mime_type: String,
+}
+
+// A directory in iCloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<chrono::DateTime<Utc>>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+impl Folder {
+ #[warn(dead_code)]
Review Comment:
doit
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,648 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, with_error_response_context, AsyncBody, HttpClient,
+ IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud signer");
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.field("region", &self.region);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ #[warn(dead_code)]
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple iCloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple iCloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ if let Some(region) = &self.region {
+ request = request.header("Origin", region);
+ request = request.header("Referer", (region.to_string() + "/").as_str());
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) = response.headers().get(ACCOUNT_COUNTRY_HEADER) {
+ self.data.account_country =
+ Some(String::from(account_country.to_str().unwrap()));
+ }
+ if let Some(session_id) = response.headers().get(SESSION_ID_HEADER) {
+ self.data.session_id = Some(String::from(session_id.to_str().unwrap()));
+ }
+ if let Some(session_token) = response.headers().get(SESSION_TOKEN_HEADER) {
Review Comment:
doit
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+ d.field("region", &self.region);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+/// [IcloudDrive](https://www.icloud.com/iclouddrive/) service support.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct IcloudBuilder {
+ pub config: IcloudConfig,
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for IcloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+
+ d.field("config", &self.config);
+ d.finish_non_exhaustive()
+ }
+}
+
+impl IcloudBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.config.root = Some(root.to_string());
+ self
+ }
+
+ /// Your Apple id
+ ///
+ /// It is required. your Apple login email, e.g. `example@gmail.com`
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.config.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id password
+ ///
+ /// It is required. your icloud login password, e.g. `password`
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.config.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ /// Trust token and ds_web_auth_token is used for temporary access to the IcloudDrive API.
+ /// Authenticate using session token
+ /// You can get more information from [pyicloud](https://github.com/picklepete/pyicloud/tree/master?tab=readme-ov-file#authentication)
+ /// or [iCloud-API](https://github.com/MauriceConrad/iCloud-API?tab=readme-ov-file#getting-started)
+
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453307921
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,648 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, with_error_response_context, AsyncBody, HttpClient,
+ IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud signer");
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.field("region", &self.region);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ #[warn(dead_code)]
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple iCloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple iCloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ if let Some(region) = &self.region {
+ request = request.header("Origin", region);
+ request = request.header("Referer", (region.to_string() + "/").as_str());
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) = response.headers().get(ACCOUNT_COUNTRY_HEADER) {
+ self.data.account_country =
+ Some(String::from(account_country.to_str().unwrap()));
+ }
+ if let Some(session_id) = response.headers().get(SESSION_ID_HEADER) {
+ self.data.session_id = Some(String::from(session_id.to_str().unwrap()));
+ }
+ if let Some(session_token) = response.headers().get(SESSION_TOKEN_HEADER) {
+ self.data.session_token = Some(String::from(session_token.to_str().unwrap()));
+ }
+
+ if let Some(scnt) = response.headers().get(SCNT_HEADER) {
+ self.data.scnt = Some(String::from(scnt.to_str().unwrap()));
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ #[warn(clippy::single_char_pattern)]
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453306200
##########
.typos.toml:
##########
@@ -23,10 +23,13 @@
"hellow" = "hellow"
# Showed up in examples.
"thw" = "thw"
+# Showed up in test.
+"hsa" = "hsa"
[files]
extend-exclude = [
# Generated SSH Keys.
"fixtures/sftp/test_ssh_key",
"fixtures/sftp/test_ssh_key.pub",
+ "core/src/services/icloud/core.rs"
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "JasCodes (via GitHub)" <gi...@apache.org>.
JasCodes commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1891252666
@bokket Awesome effort, hope this gets merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1452161272
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
Review Comment:
ref:https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392C1-L392C1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1892572854
> > Hi, please install `typos` and make our types check happy first.
>
> If `hsa` is not a typo, please add it into the ignore list with comment to explain it.
Looks well, let's continue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453306874
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
Review Comment:
LGTM
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::services::icloud::core::{parse_error, IcloudSigner, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::Client;
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub region: Option<String>,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("trust_token", &self.trust_token);
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #3980:
URL: https://github.com/apache/opendal/pull/3980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456191322
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
Review Comment:
Remove not needed comments.
##########
core/src/services/icloud/drive.rs:
##########
Review Comment:
Merge this file into `core.rs`
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
Review Comment:
How about naming this struct to `IcloudCore` to keep align with other services and avoid conflicts with so many `Client` in our project.
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple icloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ // You can get more information from [apple.com](https://support.apple.com/en-us/111754)
+ if self.is_china_mainland {
+ request = request.header("Origin", "https://www.icloud.com.cn");
+ request = request.header("Referer", "https://www.icloud.com.cn/");
+ } else {
+ request = request.header("Origin", "https://www.icloud.com");
+ request = request.header("Referer", "https://www.icloud.com/");
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) =
+ parse_header_to_str(response.headers(), ACCOUNT_COUNTRY_HEADER)?
+ {
+ self.data.account_country = Some(account_country.to_string());
+ }
+
+ if let Some(session_id) =
+ parse_header_to_str(response.headers(), SESSION_ID_HEADER)?
+ {
+ self.data.session_id = Some(session_id.to_string());
+ }
+ if let Some(session_token) =
+ parse_header_to_str(response.headers(), SESSION_TOKEN_HEADER)?
+ {
+ self.data.session_token = Some(session_token.to_string());
+ }
+
+ if let Some(scnt) = parse_header_to_str(response.headers(), SCNT_HEADER)? {
+ self.data.scnt = Some(scnt.to_string());
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ if let Some(cookie) = value.to_str().unwrap().split(';').next() {
+ if let Some((key, value)) = cookie.split_once('=') {
+ self.data
+ .cookies
+ .insert(String::from(key), String::from(value));
+ }
+ }
+ }
+ }
+ match response.status() {
+ StatusCode::UNAUTHORIZED => Err(parse_error(response).await?),
+ _ => Ok(response),
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized request",
+ )),
+ }
+ }
+}
+
+pub async fn parse_error(resp: Response<IncomingAsyncBody>) -> Result<Error> {
+ let (parts, body) = resp.into_parts();
+ let bs = body.bytes().await?;
+
+ let mut kind = match parts.status.as_u16() {
+ //status:421 Misdirected Request
+ 421 | 450 | 500 => ErrorKind::NotFound,
+ 401 => ErrorKind::Unexpected,
+ _ => ErrorKind::Unexpected,
+ };
+
+ let (message, icloud_err) = serde_json::from_reader::<_, IcloudError>(bs.clone().reader())
+ .map(|icloud_err| (format!("{icloud_err:?}"), Some(icloud_err)))
+ .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None));
+
+ if let Some(icloud_err) = &icloud_err {
+ kind = match icloud_err.status_code.as_str() {
+ "NOT_FOUND" => ErrorKind::NotFound,
+ "PERMISSION_DENIED" => ErrorKind::PermissionDenied,
+ _ => ErrorKind::Unexpected,
+ }
+ }
+
+ let mut err = Error::new(kind, &message);
+
+ err = with_error_response_context(err, parts);
+
+ Ok(err)
+}
+
+#[derive(Default, Debug, Deserialize)]
+#[allow(dead_code)]
+struct IcloudError {
+ status_code: String,
+ message: String,
+}
+
+#[derive(Default, Deserialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudWebservicesResponse {
+ #[serde(default)]
+ pub hsa_challenge_required: bool,
+ #[serde(default)]
+ pub hsa_trusted_browser: bool,
+ pub webservices: Webservices,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Webservices {
+ pub drivews: Drivews,
+ pub docws: Docws,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Drivews {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Docws {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<IcloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudObject {
+ #[serde(rename = "document_id")]
Review Comment:
Why we need those `rename` and `rename_all = "camelCase"`?
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+// A node within the icloud Drive filesystem.
+#[derive(Clone)]
+pub enum DriveNode {
+ Folder(Folder),
+}
+
+impl DriveNode {
+ fn new_root(value: &IcloudRoot) -> Result<DriveNode> {
+ Ok(DriveNode::Folder(Folder {
+ id: Some(value.drivewsid.to_string()),
+ name: value.name.to_string(),
+ date_created: Some(value.date_created.clone()),
+ items: value.items.clone(),
+ mime_type: "Folder".to_string(),
+ }))
+ }
+
+ pub fn items(&self) -> Option<Vec<IcloudItem>> {
+ match self {
+ DriveNode::Folder(folder) => Some(folder.items.clone()),
+ }
+ }
+}
+
+impl std::fmt::Display for DriveNode {
Review Comment:
We don't need Displat.
##########
core/src/services/icloud/client.rs:
##########
Review Comment:
Merge this file into `core.rs`
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
+}
+
+impl IcloudPathQuery {
+ pub fn new(client: HttpClient, core: Arc<Mutex<IcloudSigner>>) -> Self {
+ IcloudPathQuery { client, core }
+ }
+}
+
+#[async_trait]
+impl PathQuery for IcloudPathQuery {
+ async fn root(&self) -> Result<String> {
+ Ok("FOLDER::com.apple.CloudDocs::root".to_string())
+ }
+
+ // Retrieves the root directory within the icloud Drive.
+ async fn query(&self, parent_id: &str, name: &str) -> Result<Option<String>> {
+ let mut core = self.core.lock().await;
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .clone()
+ .url;
+
+ let uri = format!("{}/retrieveItemDetailsInFolders", drive_url);
+
+ let body = json!([
+ {
+ "drivewsid": parent_id,
+ "partialData": false
+ }
+ ])
+ .to_string();
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = core.sign(Method::POST, uri, async_body).await?;
+
+ if response.status() == StatusCode::OK {
+ let body = &response.into_body().bytes().await?;
+
+ let root: Vec<IcloudRoot> =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ let node = &root[0];
+
+ let id = match node.items.iter().find(|it| it.name == name) {
+ Some(it) => Ok(Some(it.drivewsid.clone())),
+ None => Ok(None),
+ }?;
+ Ok(id)
+ } else {
+ Err(parse_error(response).await?)
+ }
+ }
+
+ async fn create_dir(&self, parent_id: &str, name: &str) -> Result<String> {
+ let mut core = self.core.lock().await;
+ let client_id = core.get_client_id();
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .url
+ .clone();
+ // https://p219-drivews.icloud.com.cn:443
Review Comment:
Please don't leave unused comments.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,337 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::*;
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+
+use super::client::{Client, IcloudPathQuery};
+use super::core::{parse_error, IcloudSigner, SessionData};
+
+/// Config for icloud services support.
+#[derive(Default, Deserialize)]
+#[serde(default)]
+#[non_exhaustive]
+pub struct IcloudConfig {
+ /// root of this backend.
+ ///
+ /// All operations will happen under this root.
+ ///
+ /// default to `/` if not set.
+ pub root: Option<String>,
+ /// apple_id of this backend.
+ ///
+ /// apple_id must be full, mostly like `example@gmail.com`.
+ pub apple_id: Option<String>,
+ /// password of this backend.
+ ///
+ /// password must be full.
+ pub password: Option<String>,
+
+ /// Session
+ ///
+ /// token must be valid.
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ /// enable the china origin
+ /// China region `origin` Header needs to be set to "https://www.icloud.com.cn".
+ ///
+ /// otherwise Apple server will return 302.
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudConfig {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+ d.field("root", &self.root);
+ d.field("is_china_mainland", &self.is_china_mainland);
+ d.finish_non_exhaustive()
+ }
+}
+
+/// [IcloudDrive](https://www.icloud.com/iclouddrive/) service support.
+#[doc = include_str!("docs.md")]
+#[derive(Default)]
+pub struct IcloudBuilder {
+ /// icloud config for web session request
+ pub config: IcloudConfig,
+ /// Specify the http client that used by this service.
+ ///
+ /// # Notes
+ ///
+ /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed
+ /// during minor updates.
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for IcloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("IcloudBuilder");
+
+ d.field("config", &self.config);
+ d.finish_non_exhaustive()
+ }
+}
+
+impl IcloudBuilder {
+ /// Set root of this backend.
+ ///
+ /// All operations will happen under this root.
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.config.root = if root.is_empty() {
+ None
+ } else {
+ Some(root.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id
+ ///
+ /// It is required. your Apple login email, e.g. `example@gmail.com`
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.config.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ /// Your Apple id password
+ ///
+ /// It is required. your icloud login password, e.g. `password`
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.config.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ /// Trust token and ds_web_auth_token is used for temporary access to the icloudDrive API.
+ ///
+ /// Authenticate using session token
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
+ self.config.trust_token = if trust_token.is_empty() {
+ None
+ } else {
+ Some(trust_token.to_string())
+ };
+
+ self
+ }
+ /// ds_web_auth_token must be set in Session
+ ///
+ /// Avoid Two Factor Authentication
+ pub fn ds_web_auth_token(&mut self, ds_web_auth_token: &str) -> &mut Self {
+ self.config.ds_web_auth_token = if ds_web_auth_token.is_empty() {
+ None
+ } else {
+ Some(ds_web_auth_token.to_string())
+ };
+
+ self
+ }
+ /// Enable the china origin
+ /// For China, use "https://www.icloud.com.cn"
+ /// For Other region, use "https://www.icloud.com"
+ pub fn is_china_mainland(&mut self, is_china_mainland: bool) -> &mut Self {
+ self.config.is_china_mainland = is_china_mainland;
+ self
+ }
+
+ /// Specify the http client that used by this service.
+ ///
+ /// # Notes
+ ///
+ /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed
+ /// during minor updates.
+ pub fn http_client(&mut self, client: HttpClient) -> &mut Self {
+ self.http_client = Some(client);
+ self
+ }
+}
+
+impl Builder for IcloudBuilder {
+ const SCHEME: Scheme = Scheme::Icloud;
+ type Accessor = IcloudBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let config = IcloudConfig::deserialize(ConfigDeserializer::new(map))
+ .expect("config deserialize must succeed");
+ IcloudBuilder {
+ config,
+ http_client: None,
+ }
+ }
+
+ fn build(&mut self) -> Result<Self::Accessor> {
+ let root = normalize_root(&self.config.root.take().unwrap_or_default());
+
+ let apple_id = match &self.config.apple_id {
+ Some(apple_id) => Ok(apple_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "apple_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let password = match &self.config.password {
+ Some(password) => Ok(password.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let ds_web_auth_token = match &self.config.ds_web_auth_token {
+ Some(ds_web_auth_token) => Ok(ds_web_auth_token.clone()),
+ None => Err(
+ Error::new(ErrorKind::ConfigInvalid, "ds_web_auth_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud),
+ ),
+ }?;
+
+ let trust_token = match &self.config.trust_token {
+ Some(trust_token) => Ok(trust_token.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "trust_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ debug!(
+ "Icloud backend is_china_mainland {}",
+ &self.config.is_china_mainland
+ );
+
+ let client = if let Some(client) = self.http_client.take() {
+ client
+ } else {
+ HttpClient::new().map_err(|err| {
+ err.with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)
+ })?
+ };
+
+ let session_data = SessionData::new();
+
+ let core = IcloudSigner {
+ client: client.clone(),
+ data: session_data,
+ apple_id,
+ password,
+ trust_token: Some(trust_token),
+ ds_web_auth_token: Some(ds_web_auth_token),
+ is_china_mainland: self.config.is_china_mainland,
+ };
+
+ let core = Arc::new(Mutex::new(core));
+ Ok(IcloudBackend {
+ client: Arc::new(Client {
+ core: core.clone(),
+ root,
+ path_cache: PathCacher::new(IcloudPathQuery::new(client, core.clone())).with_lock(),
Review Comment:
Does iCloud support having files with the same name? If not, there's no need to enable the lock feature.
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
Review Comment:
Please use `signer` as field name to avoid confused with `core`.
##########
core/src/services/icloud/core.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::Buf;
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::{
+ new_json_deserialize_error, parse_header_to_str, with_error_response_context, AsyncBody,
+ HttpClient, IncomingAsyncBody,
+};
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ // This code inspire from
+ // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+#[derive(Clone)]
+pub struct ServiceInfo {
+ pub url: String,
+}
+
+#[derive(Clone)]
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub struct IcloudSigner {
+ pub client: HttpClient,
+
+ pub data: SessionData,
+ pub apple_id: String,
+ pub password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+ pub is_china_mainland: bool,
+}
+
+impl Debug for IcloudSigner {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("icloud signer");
+ de.field("is_china_mainland", &self.is_china_mainland);
+ de.finish()
+ }
+}
+
+impl IcloudSigner {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ // create_dir could use client_id
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl IcloudSigner {
+ pub async fn signer(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.sign(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: IcloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required {
+ if auth_info.hsa_trusted_browser {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple icloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl IcloudSigner {
+ pub async fn sign(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
+ let mut request = Request::builder().method(method).uri(uri);
+
+ request = request.header(OAUTH_STATE_HEADER, self.data.oauth_state.clone());
+
+ if let Some(session_id) = &self.data.session_id {
+ request = request.header(SESSION_ID_HEADER, session_id);
+ }
+ if let Some(scnt) = &self.data.scnt {
+ request = request.header(SCNT_HEADER, scnt);
+ }
+
+ // China region
+ // ("Origin", "https://www.icloud.com.cn")
+ // ("Referer", "https://www.icloud.com.cn/")
+ // You can get more information from [apple.com](https://support.apple.com/en-us/111754)
+ if self.is_china_mainland {
+ request = request.header("Origin", "https://www.icloud.com.cn");
+ request = request.header("Referer", "https://www.icloud.com.cn/");
+ } else {
+ request = request.header("Origin", "https://www.icloud.com");
+ request = request.header("Referer", "https://www.icloud.com/");
+ }
+
+ if !self.data.cookies.is_empty() {
+ let cookies: Vec<String> = self
+ .data
+ .cookies
+ .iter()
+ .map(|(k, v)| format!("{}={}", k, v))
+ .collect();
+ request = request.header(header::COOKIE, cookies.as_slice().join("; "));
+ }
+
+ if let Some(headers) = request.headers_mut() {
+ headers.insert("Content-Type", "application/json".parse().unwrap());
+ headers.insert("Accept", "*/*".parse().unwrap());
+ for (key, value) in AUTH_HEADERS {
+ headers.insert(key, value.parse().unwrap());
+ }
+ }
+
+ match self.client.send(request.body(body).unwrap()).await {
+ Ok(response) => {
+ if let Some(account_country) =
+ parse_header_to_str(response.headers(), ACCOUNT_COUNTRY_HEADER)?
+ {
+ self.data.account_country = Some(account_country.to_string());
+ }
+
+ if let Some(session_id) =
+ parse_header_to_str(response.headers(), SESSION_ID_HEADER)?
+ {
+ self.data.session_id = Some(session_id.to_string());
+ }
+ if let Some(session_token) =
+ parse_header_to_str(response.headers(), SESSION_TOKEN_HEADER)?
+ {
+ self.data.session_token = Some(session_token.to_string());
+ }
+
+ if let Some(scnt) = parse_header_to_str(response.headers(), SCNT_HEADER)? {
+ self.data.scnt = Some(scnt.to_string());
+ }
+
+ for (key, value) in response.headers() {
+ if key == header::SET_COOKIE {
+ if let Some(cookie) = value.to_str().unwrap().split(';').next() {
+ if let Some((key, value)) = cookie.split_once('=') {
+ self.data
+ .cookies
+ .insert(String::from(key), String::from(value));
+ }
+ }
+ }
+ }
+ match response.status() {
+ StatusCode::UNAUTHORIZED => Err(parse_error(response).await?),
+ _ => Ok(response),
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple icloud AuthenticationFailed:Unauthorized request",
+ )),
+ }
+ }
+}
+
+pub async fn parse_error(resp: Response<IncomingAsyncBody>) -> Result<Error> {
+ let (parts, body) = resp.into_parts();
+ let bs = body.bytes().await?;
+
+ let mut kind = match parts.status.as_u16() {
+ //status:421 Misdirected Request
+ 421 | 450 | 500 => ErrorKind::NotFound,
+ 401 => ErrorKind::Unexpected,
+ _ => ErrorKind::Unexpected,
+ };
+
+ let (message, icloud_err) = serde_json::from_reader::<_, IcloudError>(bs.clone().reader())
+ .map(|icloud_err| (format!("{icloud_err:?}"), Some(icloud_err)))
+ .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None));
+
+ if let Some(icloud_err) = &icloud_err {
+ kind = match icloud_err.status_code.as_str() {
+ "NOT_FOUND" => ErrorKind::NotFound,
+ "PERMISSION_DENIED" => ErrorKind::PermissionDenied,
+ _ => ErrorKind::Unexpected,
+ }
+ }
+
+ let mut err = Error::new(kind, &message);
+
+ err = with_error_response_context(err, parts);
+
+ Ok(err)
+}
+
+#[derive(Default, Debug, Deserialize)]
+#[allow(dead_code)]
+struct IcloudError {
+ status_code: String,
+ message: String,
+}
+
+#[derive(Default, Deserialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudWebservicesResponse {
+ #[serde(default)]
+ pub hsa_challenge_required: bool,
+ #[serde(default)]
+ pub hsa_trusted_browser: bool,
+ pub webservices: Webservices,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Webservices {
+ pub drivews: Drivews,
+ pub docws: Docws,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Drivews {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Deserialize, Default, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct Docws {
+ #[serde(rename = "pcsRequired")]
+ pub pcs_required: bool,
+ pub status: String,
+ pub url: Option<String>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<IcloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct IcloudObject {
+ #[serde(rename = "document_id")]
+ pub document_id: String,
+ #[serde(rename = "item_id")]
+ pub item_id: String,
+ #[serde(rename = "owner_dsid")]
+ pub owner_dsid: i64,
+ #[serde(rename = "data_token")]
+ pub data_token: DataToken,
+ #[serde(rename = "double_etag")]
+ pub double_etag: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct DataToken {
+ pub url: String,
+ pub token: String,
+ pub signature: String,
+ #[serde(rename = "wrapping_key")]
+ pub wrapping_key: String,
+ #[serde(rename = "reference_signature")]
Review Comment:
same question
##########
core/src/services/icloud/client.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use http::{Method, Response, StatusCode};
+use serde_json::json;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::*;
+use crate::{Error, ErrorKind, Result};
+
+use super::core::{parse_error, IcloudCreateFolder, IcloudItem, IcloudRoot, IcloudSigner};
+use super::drive::DriveService;
+
+pub struct Client {
+ pub core: Arc<Mutex<IcloudSigner>>,
+ pub root: String,
+ pub path_cache: PathCacher<IcloudPathQuery>,
+}
+
+impl Debug for Client {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("IcloudClient");
+ de.field("root", &self.root);
+ de.finish()
+ }
+}
+
+pub struct IcloudPathQuery {
+ pub client: HttpClient,
+ pub core: Arc<Mutex<IcloudSigner>>,
+}
+
+impl IcloudPathQuery {
+ pub fn new(client: HttpClient, core: Arc<Mutex<IcloudSigner>>) -> Self {
+ IcloudPathQuery { client, core }
+ }
+}
+
+#[async_trait]
+impl PathQuery for IcloudPathQuery {
+ async fn root(&self) -> Result<String> {
+ Ok("FOLDER::com.apple.CloudDocs::root".to_string())
+ }
+
+ // Retrieves the root directory within the icloud Drive.
+ async fn query(&self, parent_id: &str, name: &str) -> Result<Option<String>> {
+ let mut core = self.core.lock().await;
+ let drive_url = core
+ .get_service_info(String::from("drive"))
+ .unwrap()
+ .clone()
+ .url;
+
+ let uri = format!("{}/retrieveItemDetailsInFolders", drive_url);
Review Comment:
I'm concerned that we're scanning the entire directory just to retrieve a single file's note ID.
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
Review Comment:
We don't need to impl Iterator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1895550311
`icloud` will only list the directory, so we need to let the `path_cache` support to return `parent_id` instead of not only returning `self_id`.
The current cache : `path->parent_id`,return `self_id`.
The refactored cache: `path->self_id`,return `self_id`.The interface `insert` is not exposed.
```
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:a5e90ada-46b4-4f83-ae75-3fe1bce01ca3
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:Some("FILE::com.apple.CloudDocs::3DE26BDF-DC8F-4DDA-98C0-24E52F75587C") | name:normal_file.txt
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:Some("FILE::com.apple.CloudDocs::468C2835-A0BF-4C7A-B615-FE939C0C9A36") | name:special_file !@#$%^&()_+-=;',.txt
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:73a1d5b6-d023-47f7-ac96-6362b2217925
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:fff769a8-132e-48ca-b3ec-5b18d4237530
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:ba3996a2-8c0c-4b42-a65b-e8f1bb0efc15
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
parent_id:FOLDER::com.apple.CloudDocs::root| path_cache:None | name:f610d966-0256-468b-88ad-5d49098a5321/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1457027971
##########
core/src/services/icloud/drive.rs:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1457027169
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+// A node within the icloud Drive filesystem.
+#[derive(Clone)]
+pub enum DriveNode {
Review Comment:
done
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
Review Comment:
done
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+// A node within the icloud Drive filesystem.
+#[derive(Clone)]
+pub enum DriveNode {
+ Folder(Folder),
+}
+
+impl DriveNode {
+ fn new_root(value: &IcloudRoot) -> Result<DriveNode> {
+ Ok(DriveNode::Folder(Folder {
+ id: Some(value.drivewsid.to_string()),
+ name: value.name.to_string(),
+ date_created: Some(value.date_created.clone()),
+ items: value.items.clone(),
+ mime_type: "Folder".to_string(),
+ }))
+ }
+
+ pub fn items(&self) -> Option<Vec<IcloudItem>> {
+ match self {
+ DriveNode::Folder(folder) => Some(folder.items.clone()),
+ }
+ }
+}
+
+impl std::fmt::Display for DriveNode {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3980:
URL: https://github.com/apache/opendal/pull/3980#issuecomment-1899645939
Hello @bokket, could you address the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1891256478
> @bokket Awesome effort, hope this gets merged
just achieve `Read` and `state` method.`write` need `upload` is a little complicated.Maybe can make up next month
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1891277546
> Hi, please install `typos` and make our types check happy first.
If `hsa` is not a typo, please add it into the ignore list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456226942
##########
core/src/services/icloud/drive.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use http::header::{IF_MATCH, IF_NONE_MATCH};
+use http::{Method, Request, Response, StatusCode};
+use log::debug;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, IncomingAsyncBody, OpRead};
+use crate::Result;
+
+use super::core::{parse_error, IcloudItem, IcloudObject, IcloudRoot, IcloudSigner};
+
+#[derive(Clone)]
+pub struct File {
+ pub id: Option<String>,
+ pub name: String,
+ pub size: u64,
+ pub date_created: Option<String>,
+ pub date_modified: Option<String>,
+ pub mime_type: String,
+}
+
+// A directory in icloud Drive.
+#[derive(Clone)]
+pub struct Folder {
+ pub id: Option<String>,
+ pub name: String,
+ pub date_created: Option<String>,
+ pub items: Vec<IcloudItem>,
+ pub mime_type: String,
+}
+
+pub struct FolderIter<'a> {
+ current: std::slice::Iter<'a, IcloudItem>,
+}
+
+impl<'a> Iterator for FolderIter<'a> {
+ type Item = &'a IcloudItem;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ self.current.next()
+ }
+}
+
+// A node within the icloud Drive filesystem.
+#[derive(Clone)]
+pub enum DriveNode {
Review Comment:
Seems we are not using this part of code. How about removing them all in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1891265355
Hi, please install `typos` and make our types check happy first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1451911232
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
Review Comment:
use `super` instead of `crate::services::icloud`.
##########
core/src/services/icloud/client.rs:
##########
Review Comment:
We can rename this file to `core` for consistency with other services.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
+ self.trust_token = if trust_token.is_empty() {
+ None
+ } else {
+ Some(trust_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn ds_web_auth_token(&mut self, ds_web_auth_token: &str) -> &mut Self {
+ self.ds_web_auth_token = if ds_web_auth_token.is_empty() {
+ None
+ } else {
+ Some(ds_web_auth_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn http_client(&mut self, client: HttpClient) -> &mut Self {
+ self.http_client = Some(client);
+ self
+ }
+}
+
+impl Builder for iCloudBuilder {
+ const SCHEME: Scheme = Scheme::Icloud;
+ type Accessor = iCloudBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = Self::default();
+
+ map.get("root").map(|v| builder.root(v));
Review Comment:
Please add `IcloudConfig` instead of implement from_map by hand.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
Review Comment:
All public API should have API docs.
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub(crate) url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4().to_string()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct Session {
+ pub(crate) data: SessionData,
+ pub(crate) client: HttpClient,
+ pub(crate) apple_id: String,
+ pub(crate) password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+}
+
+impl Debug for Session {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud Session");
+ de.field("apple_id", &self.apple_id);
+ de.field("password", &self.password);
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.finish()
+ }
+}
+
+impl Session {
Review Comment:
`Session` is more like `Signer` in other services.
##########
core/src/services/mod.rs:
##########
@@ -113,6 +113,11 @@ mod ipmfs;
#[cfg(feature = "services-ipmfs")]
pub use ipmfs::Ipmfs;
+#[cfg(feature = "services-icloud")]
+mod icloud;
+#[cfg(feature = "services-icloud")]
+pub use icloud::iCloud;
Review Comment:
Please use `icloud::Icloud` instead.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
Review Comment:
Please avoid including sensitive information in the `Debug` logs.
##########
core/src/services/icloud/graph_model.rs:
##########
Review Comment:
Please merge into `core.rs`
##########
core/src/services/icloud/webservices.rs:
##########
Review Comment:
Please merge into `core.rs`
##########
core/src/services/icloud/session.rs:
##########
Review Comment:
Please merge into `core.rs`
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
Review Comment:
It's better to provide a doc on how to fetch `trust_token` and `ds_web_auth_token`.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
+ self.trust_token = if trust_token.is_empty() {
+ None
+ } else {
+ Some(trust_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn ds_web_auth_token(&mut self, ds_web_auth_token: &str) -> &mut Self {
+ self.ds_web_auth_token = if ds_web_auth_token.is_empty() {
+ None
+ } else {
+ Some(ds_web_auth_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn http_client(&mut self, client: HttpClient) -> &mut Self {
+ self.http_client = Some(client);
+ self
+ }
+}
+
+impl Builder for iCloudBuilder {
+ const SCHEME: Scheme = Scheme::Icloud;
+ type Accessor = iCloudBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = Self::default();
+
+ map.get("root").map(|v| builder.root(v));
+ map.get("apple_id").map(|v| builder.apple_id(v));
+ map.get("password").map(|v| builder.password(v));
+ map.get("trust_token").map(|v| builder.trust_token(v));
+ map.get("ds_web_auth_token")
+ .map(|v| builder.ds_web_auth_token(v));
+
+ builder
+ }
+
+ fn build(&mut self) -> Result<Self::Accessor> {
+ let root = normalize_root(&self.root.take().unwrap_or_default());
+
+ let apple_id = match &self.apple_id {
+ Some(apple_id) => Ok(apple_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "apple_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let password = match &self.password {
+ Some(password) => Ok(password.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let ds_web_auth_token = match &self.ds_web_auth_token {
+ Some(ds_web_auth_token) => Ok(ds_web_auth_token.clone()),
+ None => Err(
+ Error::new(ErrorKind::ConfigInvalid, "ds_web_auth_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud),
+ ),
+ }?;
+
+ let client = if let Some(client) = self.http_client.take() {
+ client
+ } else {
+ HttpClient::new().map_err(|err| {
+ err.with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)
+ })?
+ };
+
+ let trust_token = match &self.trust_token {
+ Some(trust_token) => Ok(trust_token.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "trust_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+ debug!("iCloud backend use trust_token {}", &trust_token);
+
+ let session_data = SessionData::new();
+
+ Ok(iCloudBackend {
+ client: Arc::new(Client {
+ session: Arc::new(Mutex::new(Session {
+ data: session_data,
+ client,
+ apple_id,
+ password,
+ trust_token: Some(trust_token),
+ ds_web_auth_token: Some(ds_web_auth_token),
+ })),
+ root,
+ path_cache: Arc::new(Default::default()),
+ }),
+ })
+ }
+}
+
+#[derive(Debug, Clone)]
+pub struct iCloudBackend {
+ client: Arc<Client>,
+}
+
+#[async_trait]
+impl Accessor for iCloudBackend {
+ type Reader = IncomingAsyncBody;
+ type BlockingReader = ();
+ type Writer = ();
+ type BlockingWriter = ();
+ type Lister = ();
+ type BlockingLister = ();
+
+ fn info(&self) -> AccessorInfo {
+ let mut ma = AccessorInfo::default();
+ ma.set_scheme(Scheme::Icloud)
+ .set_root(&self.client.root)
+ .set_native_capability(Capability {
+ stat: true,
+ read: true,
+ ..Default::default()
+ });
+ ma
+ }
+
+ async fn stat(&self, path: &str, _: OpStat) -> Result<RpStat> {
+ // iCloud get the filename by id, instead obtain the metadata by filename
+ if path == "/" {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ let node = self.client.stat(path).await?;
+
+ let mut meta = Metadata::new(match node.type_field.as_str() {
+ "FOLDER" => EntryMode::DIR,
+ _ => EntryMode::FILE,
+ });
+
+ if meta.mode() == EntryMode::DIR || path.ends_with('/') {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ meta = meta.with_content_length(node.size);
+
+ let last_modified = parse_datetime_from_rfc3339(&node.date_modified).map_err(|e| {
Review Comment:
All functions provided by `opendal::raw` handle errors correctly, so you can simply use `?` here instead.
##########
core/src/services/icloud/graph_model.rs:
##########
@@ -0,0 +1,329 @@
+use serde::{Deserialize, Serialize};
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<iCloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+impl std::fmt::Display for iCloudRoot {
Review Comment:
Please don't implement `Display` by hand. Ideally, we won't call this.
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
Review Comment:
Please keep the same naming style: `IcloudBuilder`.
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
Review Comment:
Where does this key come from?
##########
core/src/services/icloud/graph_model.rs:
##########
@@ -0,0 +1,329 @@
+use serde::{Deserialize, Serialize};
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudRoot {
+ #[serde(default)]
+ pub asset_quota: i64,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub direct_children_count: i64,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: i64,
+ pub items: Vec<iCloudItem>,
+ pub name: String,
+ pub number_of_items: i64,
+ pub status: String,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudItem {
+ #[serde(default)]
+ pub asset_quota: Option<i64>,
+ #[serde(default)]
+ pub date_created: String,
+ #[serde(default)]
+ pub date_modified: String,
+ #[serde(default)]
+ pub direct_children_count: Option<i64>,
+ pub docwsid: String,
+ pub drivewsid: String,
+ pub etag: String,
+ #[serde(default)]
+ pub file_count: Option<i64>,
+ #[serde(rename = "item_id")]
+ pub item_id: Option<String>,
+ pub name: String,
+ pub parent_id: String,
+ #[serde(default)]
+ pub size: u64,
+ #[serde(rename = "type")]
+ pub type_field: String,
+ pub zone: String,
+ #[serde(default)]
+ pub max_depth: Option<String>,
+ #[serde(default)]
+ pub is_chained_to_parent: Option<bool>,
+}
+
+impl std::fmt::Display for iCloudRoot {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+ write!(
+ f,
+ "iCloudRoot \
+ docwsid:{}\
+ drivewsid:{}\
+ items:{:?}\
+ name:{}\
+ type_field:{}\
+ zone:{}\
+ ",
+ self.docwsid, self.drivewsid, self.items, self.name, self.type_field, self.zone
+ )
+ }
+}
+impl std::fmt::Display for iCloudItem {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+ write!(
+ f,
+ "iCloudItem \
+ docwsid:{}\
+ drivewsid:{}\
+ items_id:{:?}\
+ name:{}\
+ type_field:{}\
+ zone:{}\
+ ",
+ self.docwsid, self.drivewsid, self.item_id, self.name, self.type_field, self.zone
+ )
+ }
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudObject {
+ #[serde(rename = "document_id")]
+ pub document_id: String,
+ #[serde(rename = "item_id")]
+ pub item_id: String,
+ #[serde(rename = "owner_dsid")]
+ pub owner_dsid: i64,
+ #[serde(rename = "data_token")]
+ pub data_token: DataToken,
+ #[serde(rename = "double_etag")]
+ pub double_etag: String,
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct DataToken {
+ pub url: String,
+ pub token: String,
+ pub signature: String,
+ #[serde(rename = "wrapping_key")]
+ pub wrapping_key: String,
+ #[serde(rename = "reference_signature")]
+ pub reference_signature: String,
+}
+
+impl std::fmt::Display for iCloudObject {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+ write!(
+ f,
+ "iCloudObject \
+ document_id:{}\
+ owner_dsid:{}\
+ item_id:{:?}\
+ double_etag:{}\
+ data_token:{:?}\
+ ",
+ self.document_id, self.owner_dsid, self.item_id, self.double_etag, self.data_token
+ )
+ }
+}
+
+impl std::fmt::Display for DataToken {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+ write!(
+ f,
+ "DataToken \
+ url:{}\
+ reference_signatured:{}\
+ token:{:?}\
+ wrapping_key:{}\
+ reference_signature:{}\
+ ",
+ self.url,
+ self.reference_signature,
+ self.token,
+ self.wrapping_key,
+ self.reference_signature
+ )
+ }
+}
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudCreateFolder {
+ pub destination_drivews_id: String,
+ pub folders: Vec<iCloudItem>,
+}
+
+impl std::fmt::Display for iCloudCreateFolder {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
+ write!(
+ f,
+ "iCloudCreateFolder \
+ destination_drivews_id:{}\
+ items:{:?}",
+ self.destination_drivews_id, self.folders
+ )
+ }
+}
+
+#[test]
Review Comment:
Please use
```rust
#[test]
mod tests { }
```
And please put test code inside this mod.
##########
core/src/services/icloud/graph_model.rs:
##########
@@ -0,0 +1,329 @@
+use serde::{Deserialize, Serialize};
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct iCloudRoot {
Review Comment:
Please keep the naming style.
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
Review Comment:
It's better to make it a config if `Origin` and `Referer` required header for requesting
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
+ self.password = if password.is_empty() {
+ None
+ } else {
+ Some(password.to_string())
+ };
+
+ self
+ }
+
+ pub fn trust_token(&mut self, trust_token: &str) -> &mut Self {
+ self.trust_token = if trust_token.is_empty() {
+ None
+ } else {
+ Some(trust_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn ds_web_auth_token(&mut self, ds_web_auth_token: &str) -> &mut Self {
+ self.ds_web_auth_token = if ds_web_auth_token.is_empty() {
+ None
+ } else {
+ Some(ds_web_auth_token.to_string())
+ };
+
+ self
+ }
+
+ pub fn http_client(&mut self, client: HttpClient) -> &mut Self {
+ self.http_client = Some(client);
+ self
+ }
+}
+
+impl Builder for iCloudBuilder {
+ const SCHEME: Scheme = Scheme::Icloud;
+ type Accessor = iCloudBackend;
+
+ fn from_map(map: HashMap<String, String>) -> Self {
+ let mut builder = Self::default();
+
+ map.get("root").map(|v| builder.root(v));
+ map.get("apple_id").map(|v| builder.apple_id(v));
+ map.get("password").map(|v| builder.password(v));
+ map.get("trust_token").map(|v| builder.trust_token(v));
+ map.get("ds_web_auth_token")
+ .map(|v| builder.ds_web_auth_token(v));
+
+ builder
+ }
+
+ fn build(&mut self) -> Result<Self::Accessor> {
+ let root = normalize_root(&self.root.take().unwrap_or_default());
+
+ let apple_id = match &self.apple_id {
+ Some(apple_id) => Ok(apple_id.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "apple_id is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let password = match &self.password {
+ Some(password) => Ok(password.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+
+ let ds_web_auth_token = match &self.ds_web_auth_token {
+ Some(ds_web_auth_token) => Ok(ds_web_auth_token.clone()),
+ None => Err(
+ Error::new(ErrorKind::ConfigInvalid, "ds_web_auth_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud),
+ ),
+ }?;
+
+ let client = if let Some(client) = self.http_client.take() {
+ client
+ } else {
+ HttpClient::new().map_err(|err| {
+ err.with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)
+ })?
+ };
+
+ let trust_token = match &self.trust_token {
+ Some(trust_token) => Ok(trust_token.clone()),
+ None => Err(Error::new(ErrorKind::ConfigInvalid, "trust_token is empty")
+ .with_operation("Builder::build")
+ .with_context("service", Scheme::Icloud)),
+ }?;
+ debug!("iCloud backend use trust_token {}", &trust_token);
+
+ let session_data = SessionData::new();
+
+ Ok(iCloudBackend {
+ client: Arc::new(Client {
+ session: Arc::new(Mutex::new(Session {
+ data: session_data,
+ client,
+ apple_id,
+ password,
+ trust_token: Some(trust_token),
+ ds_web_auth_token: Some(ds_web_auth_token),
+ })),
+ root,
+ path_cache: Arc::new(Default::default()),
+ }),
+ })
+ }
+}
+
+#[derive(Debug, Clone)]
+pub struct iCloudBackend {
+ client: Arc<Client>,
+}
+
+#[async_trait]
+impl Accessor for iCloudBackend {
+ type Reader = IncomingAsyncBody;
+ type BlockingReader = ();
+ type Writer = ();
+ type BlockingWriter = ();
+ type Lister = ();
+ type BlockingLister = ();
+
+ fn info(&self) -> AccessorInfo {
+ let mut ma = AccessorInfo::default();
+ ma.set_scheme(Scheme::Icloud)
+ .set_root(&self.client.root)
+ .set_native_capability(Capability {
+ stat: true,
+ read: true,
+ ..Default::default()
+ });
+ ma
+ }
+
+ async fn stat(&self, path: &str, _: OpStat) -> Result<RpStat> {
+ // iCloud get the filename by id, instead obtain the metadata by filename
+ if path == "/" {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ let node = self.client.stat(path).await?;
+
+ let mut meta = Metadata::new(match node.type_field.as_str() {
+ "FOLDER" => EntryMode::DIR,
+ _ => EntryMode::FILE,
+ });
+
+ if meta.mode() == EntryMode::DIR || path.ends_with('/') {
+ return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+ }
+
+ meta = meta.with_content_length(node.size);
+
+ let last_modified = parse_datetime_from_rfc3339(&node.date_modified).map_err(|e| {
+ Error::new(ErrorKind::Unexpected, "parse last modified time").set_source(e)
+ })?;
+ meta = meta.with_last_modified(last_modified);
+
+ Ok(RpStat::new(meta))
+ }
+
+ async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> {
+ let resp = self.client.read(path, &args).await?;
+ let status = resp.status();
+
+ match status {
+ StatusCode::OK | StatusCode::PARTIAL_CONTENT => {
+ let size = parse_content_length(resp.headers())?;
+ let range = parse_content_range(resp.headers())?;
+ Ok((
+ //0-1023
Review Comment:
Please remove not meaningful comments.
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+];
+
+pub struct ServiceInfo {
+ pub(crate) url: String,
+}
+
+pub struct SessionData {
+ oauth_state: String,
+ session_id: Option<String>,
+ session_token: Option<String>,
+
+ scnt: Option<String>,
+ account_country: Option<String>,
+
+ cookies: BTreeMap<String, String>,
+ webservices: HashMap<String, ServiceInfo>,
+}
+
+impl SessionData {
+ pub fn new() -> SessionData {
+ Self {
+ oauth_state: format!("auth-{}", uuid::Uuid::new_v4().to_string()).to_string(),
+ session_id: None,
+ session_token: None,
+ scnt: None,
+ account_country: None,
+ cookies: BTreeMap::new(),
+ webservices: HashMap::new(),
+ }
+ }
+}
+
+pub struct Session {
+ pub(crate) data: SessionData,
+ pub(crate) client: HttpClient,
+ pub(crate) apple_id: String,
+ pub(crate) password: String,
+
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+}
+
+impl Debug for Session {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut de = f.debug_struct("iCloud Session");
+ de.field("apple_id", &self.apple_id);
+ de.field("password", &self.password);
+ de.field("trust_token", &self.trust_token);
+ de.field("ds_web_auth_token", &self.ds_web_auth_token);
+ de.finish()
+ }
+}
+
+impl Session {
+ pub fn get_service_info(&self, name: String) -> Option<&ServiceInfo> {
+ self.data.webservices.get(&name)
+ }
+
+ pub fn get_client_id(&self) -> &String {
+ &self.data.oauth_state
+ }
+}
+
+impl Session {
+ pub async fn sign(&mut self) -> Result<()> {
+ let body = json!({
+ "accountName" : self.apple_id,
+ "password" : self.password,
+ "rememberMe": true,
+ "trustTokens": [self.trust_token.clone().unwrap()],
+ })
+ .to_string();
+
+ let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.request(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ return match status {
+ StatusCode::OK => {
+ if let Some(rscd) = response.headers().get(APPLE_RESPONSE_HEADER) {
+ let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap();
+ //409
+ if status_code != StatusCode::CONFLICT {
+ return Err(parse_error(response).await?);
+ }
+ }
+ self.authenticate().await
+ }
+ _ => Err(parse_error(response).await?),
+ };
+ }
+
+ pub async fn authenticate(&mut self) -> Result<()> {
+ let body = json!({
+ "accountCountryCode": self.data.account_country.as_ref().unwrap_or(&String::new()),
+ "dsWebAuthToken":self.ds_web_auth_token.as_ref().unwrap_or(&String::new()),
+ "extended_login": true,
+ "trustToken": self.trust_token.as_ref().unwrap_or(&String::new())
+ })
+ .to_string();
+
+ let uri = format!("{}/accountLogin", SETUP_ENDPOINT);
+
+ let async_body = AsyncBody::Bytes(bytes::Bytes::from(body));
+
+ let response = self.request(Method::POST, uri, async_body).await?;
+
+ let status = response.status();
+
+ match status {
+ StatusCode::OK => {
+ let body = &response.into_body().bytes().await?;
+ let auth_info: iCloudWebservicesResponse =
+ serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?;
+
+ if let Some(drivews_url) = &auth_info.webservices.drivews.url {
+ self.data.webservices.insert(
+ String::from("drive"),
+ ServiceInfo {
+ url: drivews_url.to_string(),
+ },
+ );
+ }
+ if let Some(docws_url) = &auth_info.webservices.docws.url {
+ self.data.webservices.insert(
+ String::from("docw"),
+ ServiceInfo {
+ url: docws_url.to_string(),
+ },
+ );
+ }
+
+ if auth_info.hsa_challenge_required == true {
+ if auth_info.hsa_trusted_browser == true {
+ Ok(())
+ } else {
+ Err(Error::new(ErrorKind::Unexpected, "Apple iCloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication"))
+ }
+ } else {
+ Ok(())
+ }
+ }
+ _ => Err(Error::new(
+ ErrorKind::Unexpected,
+ "Apple iCloud AuthenticationFailed:Unauthorized:Invalid token",
+ )),
+ }
+ }
+}
+
+impl Session {
+ pub async fn request(
+ &mut self,
+ method: Method,
+ uri: String,
+ body: AsyncBody,
+ ) -> Result<Response<IncomingAsyncBody>> {
Review Comment:
Please implement `sign(&mut req)` instead.
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ ),
+ ("X-Apple-OAuth-Client-Type", "firstPartyAuth"),
+ ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"),
+ ("X-Apple-OAuth-Require-Grant-Code", "true"),
+ ("X-Apple-OAuth-Response-Mode", "web_message"),
+ ("X-Apple-OAuth-Response-Type", "code"),
+ (
+ "X-Apple-Widget-Key",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
Review Comment:
Where does this key come from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1893918355
Hi, path cache has been added in main, maybe you want to refactor by it instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1453355652
##########
core/src/services/icloud/session.rs:
##########
@@ -0,0 +1,304 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::{BTreeMap, HashMap};
+use std::fmt::{Debug, Formatter};
+
+use http::header;
+use http::Method;
+use http::Request;
+use http::Response;
+use http::StatusCode;
+
+use crate::types::Result;
+use crate::{Error, ErrorKind};
+
+use crate::raw::oio::WriteBuf;
+use crate::raw::{new_json_deserialize_error, AsyncBody, HttpClient, IncomingAsyncBody};
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::webservices::iCloudWebservicesResponse;
+use serde_json::json;
+
+//TODO It just support cn
+//Other country shoule be https://www.icloud.com
+//Need to improve?
+const GLOBAL_HEADERS: [(&str, &str); 2] = [
+ ("Origin", "https://www.icloud.com.cn"),
+ ("Referer", "https://www.icloud.com.cn/"),
+];
+
+static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country";
+static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State";
+
+static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id";
+
+static SCNT_HEADER: &str = "scnt";
+
+static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token";
+static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth";
+static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1";
+
+static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd";
+
+const AUTH_HEADERS: [(&str, &str); 7] = [
+ (
+ "X-Apple-OAuth-Client-Id",
+ "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
Review Comment:
Please add in comments so the maintainer knows where it's from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(iCloud):Add Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1452598061
##########
core/src/services/icloud/backend.rs:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::raw::*;
+use crate::services::icloud::client::Client;
+use crate::services::icloud::error::parse_error;
+use crate::services::icloud::session::{Session, SessionData};
+use crate::*;
+use crate::{Builder, Capability, Error, ErrorKind, Scheme};
+use async_trait::async_trait;
+use http::StatusCode;
+use log::debug;
+use std::collections::HashMap;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+#[derive(Default)]
+pub struct iCloudBuilder {
+ pub root: Option<String>,
+ pub apple_id: Option<String>,
+ pub password: Option<String>,
+ pub trust_token: Option<String>,
+ pub ds_web_auth_token: Option<String>,
+
+ pub http_client: Option<HttpClient>,
+}
+
+impl Debug for iCloudBuilder {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ let mut d = f.debug_struct("iCloudBuilder");
+ d.field("root", &self.root);
+ d.field("apple_id", &self.apple_id);
+ d.field("password", &self.password);
+ d.field("trust_token", &self.trust_token);
+ d.field("ds_web_auth_token", &self.ds_web_auth_token);
+
+ d.finish_non_exhaustive()
+ }
+}
+
+impl iCloudBuilder {
+ pub fn root(&mut self, root: &str) -> &mut Self {
+ self.root = Some(root.to_string());
+ self
+ }
+
+ pub fn apple_id(&mut self, apple_id: &str) -> &mut Self {
+ self.apple_id = if apple_id.is_empty() {
+ None
+ } else {
+ Some(apple_id.to_string())
+ };
+
+ self
+ }
+
+ pub fn password(&mut self, password: &str) -> &mut Self {
Review Comment:
doit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#issuecomment-1895572607
> `icloud` will only list the directory, so we need to let the `path_cache` support to return `parent_id` instead of not only returning `self_id`.
I'm confused. To find the parent of `a/b/c`, can't we simply look up the ID of `a/b/`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(icloud): Add basic Apple iCloud Drive support [incubator-opendal]
Posted by "bokket (via GitHub)" <gi...@apache.org>.
bokket commented on code in PR #3980:
URL: https://github.com/apache/incubator-opendal/pull/3980#discussion_r1456235538
##########
core/src/services/icloud/client.rs:
##########
Review Comment:
This may make a single file `core.rs` too long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org