You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2023/12/06 07:15:22 UTC
[I] [Python] Clean up ExtensionType.__reduce__ [arrow]
AlenkaF opened a new issue, #39094:
URL: https://github.com/apache/arrow/issues/39094
### Describe the enhancement requested
Currently `__reduce__` method is defined on the `ExtensionType` class so that each subclass can be pickled and doesn't need a separate `__reduce__` method (see https://github.com/apache/arrow/pull/36170).
But this can be cleaned up. Instead of defining `__reduce__` for `ExtensionType` we can define `__reduce__` on `BaseExtensionType` and make use of the C++-level (de)serialization. See https://github.com/apache/arrow/pull/35933#issuecomment-1841014923.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-1845651262
However, that then only works for registered extension types. I don't know if we are OK with that restriction. In a Python session you could also be working with an ExtensionType that hasn't been registered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-1845685996
That's a good point, I'm not sure which is better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-1845648082
Ah, yes of course, just serialize the name and get the instance from the registry based on the name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-1845619410
The way we would do it is like in IPC: serialize the extension name and the extension serialized form.
Then on deserialization (in C++):
1. first get the extension type from its name using `GetExtensionType`
2. then deserialize the actual instance using `ExtensionType::Deserialize`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-1845614035
Quickly looking into this, but now I am unsure whether it's actually an easy change as I mentioned at https://github.com/apache/arrow/pull/35933#issuecomment-1841288493 to use `self.ext_type.Deserialize(..)`.
To be able to call the C++ `Deserialize` method, we first need to instantiate the relevant C++ ExtensionType in the cython unpickle method. But how do you do that? When you have a python instance, you can get that with `self.ext_type`. But when unpickling, you don't have an instance of the specific extension type class yet, because that's exactly what you are unpickling.
For pyarrow.ExtensionType, `__arrow_ext_deserialize__` is also a classmethod (i.e. doesn't require an instance), but the implementation here is in python and custom done by the user and can call the class constructor of itself as it sees fit. However, if we want to put similar generic logic on the BaseExtensionType class, there is no `__init__` that we can call.
But maybe I am overlooking something cc @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Clean up ExtensionType.__reduce__ [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39094:
URL: https://github.com/apache/arrow/issues/39094#issuecomment-3921988972
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org