You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2024/04/16 17:22:36 UTC
[PR] feat(python): Implement extension type support [arrow-nanoarrow]
paleolimbot opened a new pull request, #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431
Still a work in progress!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2061169372
> Do we want to go the route of a registry and having users define their own?
Great point! I'm still feeling my way through how users should interact with this.
The part where the concept of a registry is difficult to avoid (global or tightly scoped) is when doing conversion to/from Python objects with nested type support. Without a registry, how would one produce/consume a `struct<geoarrow.polygon, uuid>` without inefficiently bouncing through deeply-nested Python output or without every package that does consuming explicitly opting in to every possible extension type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type support [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2060747347
(we could of course still provide some more ergonomic access to the extension name/metadata, e.g. by detecting if those keys are present, and in that case showing them more prominent in certain reprs, or provide an easier getter property for them, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type support [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2066696388
> The part where the concept of a registry is difficult to avoid (global or tightly scoped) is when doing conversion to/from Python objects with nested type support.
The user has always cheap access to the storage type, or the raw data, and can do a custom conversion from that. For the example of Polygons, I wouldn't want some iterative approach to convert to python objects, I would just want to have access to the underlying coordinates and offsets to do a custom bulk conversion.
I personally would punt on this problem of custom conversion for now, and see what kind of needs arise (we don't even support this in pyarrow at the moment)
If some kind of registry is needed specifically for the conversion, the sqlite interface could be an inspiration? (not necessarily the exact names, because I find those a bit confusing. But the fact that you register just something for conversion from/to python, and that also doesn't necessarily need to be limited to extension types. Users might want to influence how for example a map or interval type is converted to python)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type and Schema metadata support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2067530671
> Users might want to influence how for example a map or interval type is converted to python)
Right now that's a little awkward, but possible:
```python
import nanoarrow as na
import pyarrow as pa
from nanoarrow import iterator
import decimal
decimals = pa.array([decimal.Decimal("1.234")], pa.decimal128(10, 3))
class CustomIterator(iterator.PyIterator):
def _decimal_iter(self, offset, length):
for item in super()._decimal_iter(offset, length):
yield None if item is None else float(item)
list(CustomIterator.get_iterator(decimals))
#> [1.234]
```
> I personally would punt on this problem of custom conversion for now
Agreed! I think I've scoped it now to the basic usage of `if schema.extension and schema.extension.name == ...`.
> (we don't even support this in pyarrow at the moment)
I'm not sure exactly where it is used, but you can add a custom `Scalar` subclass that overrides `as_py()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type and Schema metadata support [arrow-nanoarrow]
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#discussion_r1581480452
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -967,10 +989,50 @@ cdef class CSchemaBuilder:
if self._ptr.release == NULL:
ArrowSchemaInit(self._ptr)
+ @staticmethod
+ def copy_existing(CSchema existing_schema):
+ return CSchemaBuilder(existing_schema.__deepcopy__())
Review Comment:
nit: I'd prefer a shorter name. WDYT?
```suggestion
def copy(CSchema schema):
return CSchemaBuilder(schema.__deepcopy__())
```
##########
python/src/nanoarrow/iterator.py:
##########
@@ -160,6 +164,17 @@ def get_iterator(cls, obj, schema=None):
yield from iterator._iter1(0, array.length)
def _iter1(self, offset, length):
+ # Check for and extension type first since this isn't reflected by
Review Comment:
```suggestion
# Check for an extension type first since this isn't reflected by
```
##########
python/src/nanoarrow/schema.py:
##########
@@ -108,10 +115,73 @@ def create(obj):
return TimeUnit(obj)
+class ExtensionAccessor:
+ """Accessor for extension type parameters"""
+
+ def __init__(self, schema) -> None:
+ self._schema = schema
+
+ @property
+ def name(self) -> str:
+ """Extension name for this extension type"""
+ return self._schema._c_schema_view.extension_name
+
+ @property
+ def metadata(self) -> Union[bytes, None]:
+ """Extension metadata for this extension type if present"""
+ extension_metadata = self._schema._c_schema_view.extension_metadata
+ return extension_metadata if extension_metadata else None
+
+ @property
+ def storage(self):
+ """Storage type for this extension type"""
+ metadata = dict(self._schema.metadata.items())
+ del metadata[b"ARROW:extension:name"]
+ if b"ARROW:extension:metadata" in metadata:
+ del metadata[b"ARROW:extension:metadata"]
Review Comment:
why do we delete the objects here?
##########
python/src/nanoarrow/schema.py:
##########
@@ -108,10 +115,73 @@ def create(obj):
return TimeUnit(obj)
+class ExtensionAccessor:
+ """Accessor for extension type parameters"""
+
+ def __init__(self, schema) -> None:
+ self._schema = schema
+
+ @property
+ def name(self) -> str:
+ """Extension name for this extension type"""
+ return self._schema._c_schema_view.extension_name
+
+ @property
+ def metadata(self) -> Union[bytes, None]:
+ """Extension metadata for this extension type if present"""
+ extension_metadata = self._schema._c_schema_view.extension_metadata
+ return extension_metadata if extension_metadata else None
+
+ @property
+ def storage(self):
+ """Storage type for this extension type"""
+ metadata = dict(self._schema.metadata.items())
+ del metadata[b"ARROW:extension:name"]
+ if b"ARROW:extension:metadata" in metadata:
+ del metadata[b"ARROW:extension:metadata"]
+
+ return Schema(self._schema, metadata=metadata)
+
+
class Schema:
- """The Schema is nanoarrow's high-level data type representation whose scope maps to
- that of the ArrowSchema in the Arrow C Data interface. See :func:`schema` for class
- details.
+ """Create a nanoarrow Schema
+
+ The Schema is nanoarrow's high-level data type representation, encompasing
Review Comment:
```suggestion
The Schema is nanoarrow's high-level data type representation, encompassing
```
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -967,10 +989,50 @@ cdef class CSchemaBuilder:
if self._ptr.release == NULL:
ArrowSchemaInit(self._ptr)
+ @staticmethod
+ def copy_existing(CSchema existing_schema):
+ return CSchemaBuilder(existing_schema.__deepcopy__())
+
@staticmethod
def allocate():
return CSchemaBuilder(CSchema.allocate())
+ def clear_metadata(self):
Review Comment:
Optional: What if we made this more object oriented? e.g. `CSchemaBuilder.clear_metadata()` vs `CSchemaBuilder.metadata.clear()`. Same with `append_metadata`. Does it make sense to implement this way? If not (or its a lot of work), ignore this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type and Schema metadata support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#discussion_r1581712390
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -967,10 +989,50 @@ cdef class CSchemaBuilder:
if self._ptr.release == NULL:
ArrowSchemaInit(self._ptr)
+ @staticmethod
+ def copy_existing(CSchema existing_schema):
+ return CSchemaBuilder(existing_schema.__deepcopy__())
+
@staticmethod
def allocate():
return CSchemaBuilder(CSchema.allocate())
+ def clear_metadata(self):
Review Comment:
At this level of abstraction it would be hard (since it is basically wrapping a single C API call) but I agree that it would be a better interface!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type and Schema metadata support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2080322076
Thank you for taking a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type and Schema metadata support [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#discussion_r1581712127
##########
python/src/nanoarrow/schema.py:
##########
@@ -108,10 +115,73 @@ def create(obj):
return TimeUnit(obj)
+class ExtensionAccessor:
+ """Accessor for extension type parameters"""
+
+ def __init__(self, schema) -> None:
+ self._schema = schema
+
+ @property
+ def name(self) -> str:
+ """Extension name for this extension type"""
+ return self._schema._c_schema_view.extension_name
+
+ @property
+ def metadata(self) -> Union[bytes, None]:
+ """Extension metadata for this extension type if present"""
+ extension_metadata = self._schema._c_schema_view.extension_metadata
+ return extension_metadata if extension_metadata else None
+
+ @property
+ def storage(self):
+ """Storage type for this extension type"""
+ metadata = dict(self._schema.metadata.items())
+ del metadata[b"ARROW:extension:name"]
+ if b"ARROW:extension:metadata" in metadata:
+ del metadata[b"ARROW:extension:metadata"]
Review Comment:
I added a comment! (Removing the metadata keys is required, or else the schema will still be treated as an extension).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Implement extension type support [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #431:
URL: https://github.com/apache/arrow-nanoarrow/pull/431#issuecomment-2060743694
Do we want to go the route of a registry and having users define their own?
For nanoarrow, I would personally stick to what it in essence is: metadata (and we have the advantage that here the "data type" (ArrowSchema) of an array actually has this field metadata, and so we don't have the problem like in pyarrow that the Array object looses that information)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org