You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "TennyZhuang (via GitHub)" <gi...@apache.org> on 2024/11/15 02:00:40 UTC

[PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

TennyZhuang opened a new pull request, #5324:
URL: https://github.com/apache/opendal/pull/5324

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Partially finished #5316 
   
   # Rationale for this change
   
   <!--
   Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
   Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   Now an `Operator` can be serialized and deserialized via pickle, as a result, the `Operator` class can be used in multiprocessing context.
   
   # What changes are included in this PR?
   
   In the PR, we store the arguments for `Operator` constructor, so that we can recover the `Operator` from pickle deserialization.
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are there any user-facing changes?
   
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking-changes` label.
   -->
   
   - support pickle [de]serialization for `Operator` in python binding
   - support `Operator` to be used in multiprocessing context
   
   @Zheaoli I'm not very familiar with Python binding, PTAL for the PR, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844921776


##########
bindings/python/tests/test_pickle_types.py:
##########
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import pickle
+
+
+@pytest.mark.need_capability()
+def test_operator_pickle(service_name, operator, async_operator):
+    """
+    Test Operator's pickle serialization and deserialization.
+    """
+    serialized = pickle.dumps(operator)
+    deserialized = pickle.loads(serialized)
+    assert repr(operator) == repr(deserialized)

Review Comment:
   Sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480594543

   Will test another PR use the CI flow, just ignore it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844249728


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   I think we need add extra `__hash__` method to represent two object are equal.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844975194


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   I prefer don’t implement __hash__ and __eq__ for Operator, and remove the equal assertion based on __repr__.
   
   (schema, map) can’t identify an Operator, leave __eq__ and __empty__ empty is better. Only the ID of PyObject can identify the object.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480559706

   > I don't like the idea that:
   
   Please note that `pickle` doesn’t mean `shared`. The user can pickle the operator, close the operator, pass the bytes to another process and unpickle the operator. So we should implement pickle for every operator, both shared and unique.
   
   What we only can do is disable the tests for unique operator.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480592055

   > Please note that `shared` only indicates shared between processes but not shared between threads, it may be a little confusing if we will add backends which only support single-thread later.
   > 
   > However, I guess we will not support that in a while.
   
   Currently, we don't offer any services that aren't `Send` because `Access` requires services to be both `Send` and `Sync`. There are no plans to add such services, but we can revisit this decision if necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844975194


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   I prefer don’t implement __hash__ and __eq__ for Operator, and remove the equal assertion based on __repr__.
   
   (schema, map) can’t identify an Operator, leave __eq__ and __empty__ empty is better. Only the ID of PyObject can identify the Operator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2479601119

   > It seems like we should add a new capability `MultiProcessingAccess` to guard the behavior.
   
   SGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480580961

   > I support temporarily adding only shared capability.
   
   Would you like to add this capability, I believe we can include them in the next 0.51 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2477821934

   It seems like we should add a new capability `MultiProcessingAccess` to guard the behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480450264

   > Pickable
   
   IMO all operators can be serialized as pickle.
   
   In most Python libraries, all objects can be serialized as pickle, but it may not be identical when deserialized. It's acceptable.
   
   We need the capability only for tests. We can ignore backends which doesn't support the capability in the test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480499938

   > I'm considering adding a `shared` capability to indicate whether this storage service can be shared across different processes or even different instances.
   
   SGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844249163


##########
bindings/python/tests/test_pickle_types.py:
##########
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import pickle
+
+
+@pytest.mark.need_capability()
+def test_operator_pickle(service_name, operator, async_operator):
+    """
+    Test Operator's pickle serialization and deserialization.
+    """
+    serialized = pickle.dumps(operator)
+    deserialized = pickle.loads(serialized)
+    assert repr(operator) == repr(deserialized)

Review Comment:
   For me, I think the ideal behavior for `pickle [de]serialization` is that the object has the same behavior/operation result.
   
   So I suggest the test can be
   
   ```python
   data="abc"
   operator.write("/abc","abc")
   serialized = pickle.dumps(operator)
   del operator
   gc.collect()
   
   deserialized = pickle.loads(serialized)
   assert "abc"==deserialized.read("/abc")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480461643

   I'm considering adding a `shared` capability to indicate whether this storage service can be shared across different processes or even different instances.
   
   Based on this definition:
   
   - `memory` is not shared: it cannot be accessed by other processes.
   - `sled` is not shared: it has a unique lock that only one process can use.
   - `fs` is shared: different processes can access the same fs as long as they have it mounted.
   - `s3` is shared: obviously.
   
   I believe this capability is better than the following:
   
   - `multi_processing_access`: too long and doesn't align with exstsing capability naming style.
   - `pickable`: python only and doesn't reflect the root cause.
   
   ---
   
   I don't like the idea that:
   
   > We need the capability only for tests
   
   I view it as a design failure for OpenDAL to include something in the public API solely for testing purposes. `shared` is somewhat useful for users who want to understand what will happen if they try to share the same storage services across processes or instances. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844931726


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   how about hash the scheme and the map? 
   
   **__repr__** is used for show what the object is looking at. Maybe not a good method for identify



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480587932

   > > I support temporarily adding only shared capability.
   > 
   > Would you like to add this capability? I believe we can include them in the next 0.51 release.
   
   I can add it.
   
   Please note that `shared` only indicates shared between processes but not shared between threads, it may be a little confusing if we will add backends which only support single-thread later.
   
   However, I guess we will not support that in a while.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844249728


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   I think we need add extra __hash__ to represent two object are equal.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844931726


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   how about hash the scheme and the map? 
   
   __repr__ is used for show what the object is looking at. Maybe not a good method for identify



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844975420


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   > I prefer don’t implement **hash** and **eq** for Operator, and remove the equal assertion based on **repr**.
   
   Agreed. `ocore::Operator` doesn't implement `Eq` ahd `Hash` too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on code in PR #5324:
URL: https://github.com/apache/opendal/pull/5324#discussion_r1844921050


##########
bindings/python/src/operator.rs:
##########
@@ -184,15 +200,21 @@ impl Operator {
     }
 
     pub fn capability(&self) -> PyResult<capability::Capability> {
-        Ok(capability::Capability::new(self.0.info().full_capability()))
+        Ok(capability::Capability::new(
+            self.core.info().full_capability(),
+        ))
     }
 
     pub fn to_async_operator(&self) -> PyResult<AsyncOperator> {
-        Ok(AsyncOperator(self.0.clone().into()))
+        Ok(AsyncOperator {
+            core: self.core.clone().into(),
+            __scheme: self.__scheme.clone(),
+            __map: self.__map.clone(),
+        })
     }
 
     fn __repr__(&self) -> String {

Review Comment:
   It's hard to implement, we can't get internal information of `Operator`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480563156

   > Please note that `pickle` doesn’t mean `shared`. The user can pickle the operator, close the operator, pass the bytes to another process and unpickle the operator. So we should implement pickle for every operator, both shared and unique.
   
   That's correct. As you mentioned, we just need to avoid testing it in a shared manner (which implies that we can access the same storage service after unpickling) if this storage service doesn't `shared`.
   
   I prefer to add the `shared` capability, which I believe is a comprehensive solution. However, I'm also open to the python binding just testing against specified services like `s3`, `gcs`, and `fs` if you feel that's sufficient.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "TennyZhuang (via GitHub)" <gi...@apache.org>.
TennyZhuang commented on PR #5324:
URL: https://github.com/apache/opendal/pull/5324#issuecomment-2480566420

   In fact, there are two types of capabilities in all backends.
   
   1. shared
   Access rights can be shared, all services backends implement this, while embedded engines don't.
   
   2. send
   Access rights can be transferred
   Almost all backends implement this, except for memory backend such as dashmap.
   
   Of course, we don't need to be so strict and complicated, I support temporarily adding only shared capability.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(bindings/python): support pickle [de]serialization for Operator [opendal]

Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli merged PR #5324:
URL: https://github.com/apache/opendal/pull/5324


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org