You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "esafak (via GitHub)" <gi...@apache.org> on 2025/07/05 20:00:44 UTC
[PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
esafak opened a new pull request, #2384:
URL: https://github.com/apache/fory/pull/2384
## What does this PR do?
* Enable correct serialization/deserialization of objects using `__getstate__` and `__setstate__` when Fory's Python serialization falls back to the standard pickle mechanism.
* Modify `_registry.py` to allow `PickleSerializer` fallback for `Language.XLANG` when `require_type_registration=False`.
* Update `serializer.py` to make `PickleSerializer` inherit from `CrossLanguageCompatibleSerializer`, ensuring `xwrite` and `xread` delegate to `write` and `read`.
* Add tests in `test_serializer.py` to verify `__getstate__` and `__setstate__` behavior during Fory's pickle fallback.
## Related issues
Completes #2293, #2294
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #2384:
URL: https://github.com/apache/fory/pull/2384#issuecomment-3041217129
Languages like java/go can't deserialize binary serialized by pickle, maybe we don't need it be CrossLanguageCompatibleSerializer? And we could invoke `__getstate__/__setstate__` indrectly in fory to make fory suppoert such objects instead of fallback to pickle. In the long run, I think fory can be a drop-in replacement for pickle but with better performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #2384:
URL: https://github.com/apache/fory/pull/2384#issuecomment-3041995907
> How about this plan?
>
> In short, the idea is to leverage Fory’s native serialization for objects that support customization, gradually replacing pickle's behavior with a more transparent and cross-language-friendly approach while maintaining compatibility with edge cases.
Looks great! 👍
With this plan, fory will be able to serialize any objects in python if it can be serialized by pickle, but with fory serialization format, the serialization will be faster.
Another thing is to support local classes and lambda like cloudpickle. But we can do that after finished this plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
Posted by "esafak (via GitHub)" <gi...@apache.org>.
esafak closed pull request #2384: feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer
URL: https://github.com/apache/fory/pull/2384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
Posted by "esafak (via GitHub)" <gi...@apache.org>.
esafak commented on PR #2384:
URL: https://github.com/apache/fory/pull/2384#issuecomment-3055372629
> Another thing is to support local classes and lambda like cloudpickle.
Could you create an issue and describe how you think it should be done?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [PR] feat(python): Make PickleSerializer a CrossLanguageCompatibleSerializer [fory]
Posted by "esafak (via GitHub)" <gi...@apache.org>.
esafak commented on PR #2384:
URL: https://github.com/apache/fory/pull/2384#issuecomment-3041917723
How about this plan?
1. Define two new serializers:
- A "Stateful Serializer" for objects that support __getstate__ and __setstate__.
- A "Reduce Serializer" for objects that implement __reduce__ or __reduce_ex__.
2. For objects with __getstate__/__setstate__:
- In the serialization function, check if both methods are available.
- If yes, call __getstate__ to obtain the object state and then serialize that state using Fory's native methods.
- For deserialization, create a new instance without calling __init__, then use the deserialized state with __setstate__.
3. For objects with __reduce__/__reduce_ex__:
- Detect if these methods exist and give them higher precedence over __getstate__/__setstate__.
- Support basic cases where __reduce__ returns either just a global name or a tuple (callable, args).
- Extend to handle objects that include state or more complex information (like iterators or custom functions).
4. For handling constructors (especially for immutable types):
- Check for __getnewargs_ex__ or __getnewargs__ during instance creation.
- Serialize the constructor arguments so that during deserialization, the new instance is created with the correct arguments.
5. Testing & Backwards Compatibility:
- Implement incremental tests to cover each serialization case (basic state, __reduce__ cases, and __getnewargs*__ cases).
- Ensure that existing functionality still falls back to the Python pickle behavior when necessary, but aim to use Fory’s native XLANG format for better cross-language support and performance.
In short, the idea is to leverage Fory’s native serialization for objects that support customization, gradually replacing pickle's behavior with a more transparent and cross-language-friendly approach while maintaining compatibility with edge cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org