You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "henryharbeck (via GitHub)" <gi...@apache.org> on 2025/10/12 07:01:32 UTC
[PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
henryharbeck opened a new pull request, #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556
Implements `import_schema()` for the Polars DBAPI backend.
This means `Connection.adbc_get_table_schema`, `Cursor.adbc_execute_partitions`, `Cursor.adbc_execute_schema`, and `Cursor.adbc_prepare` can now return a PyArrow Schema or a Polars Schema. Unfortunately it does not seem like the return type can be known statically, so I haven't added any overloads. If the ambiguous return type isn't desired, please feel free to suggest alternatives or close this PR.
Support for initializing a `polars.Schema` from the `__arrow_c_schema__` interface was added in Polars [1.32.1](https://github.com/pola-rs/polars/releases/tag/py-1.32.1) (https://github.com/pola-rs/polars/pull/23879) (which was yanked, so really 1.32.2)
I've left a few comments where I'm unsure of whether some alternative code may preferred
Closes https://github.com/apache/arrow-adbc/issues/3421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2423372446
##########
python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py:
##########
@@ -179,8 +179,21 @@ def import_array_stream(
) -> typing.Any:
return polars.from_arrow(handle)
- def import_schema(self, handle: _lib.ArrowSchemaHandle) -> typing.Any:
- raise _lib.NotSupportedError("Polars does not support __arrow_c_schema__")
+ def import_schema(self, handle: _lib.ArrowSchemaHandle) -> polars.Schema:
+ def parse_version(version: str) -> tuple[int, ...]:
+ return tuple(int(v) for v in version.split("."))
+
+ # The version that Polars added support to initialize a schema via the
+ # __arrow_c_schema__ interface
+ required_version = "1.32.2"
+ polars_version = polars.__version__
+ if parse_version(polars_version) >= parse_version(required_version):
+ return polars.Schema(handle)
Review Comment:
Alternatively, could leave the version check and just `return polars.Schema(handle)` immediately.
Would raise with the below before the required version. IMO providing the user a nicer error message is preferable.
```
Traceback (most recent call last):
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/try_import_schema.py", line 11, in <module>
schema = conn.adbc_get_table_schema("test_table_schema")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/dbapi.py", line 521, in adbc_get_table_schema
return self._backend.import_schema(handle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py", line 183, in import_schema
return polars.Schema(handle)
^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/.cache/uv/archive-v0/SC8MnmcDLgKYq5uEoGc4z/lib/python3.11/site-packages/polars/schema.py", line 102, in __init__
for name, tp in input:
TypeError: 'adbc_driver_manager._lib.ArrowSchemaHandle' object is not iterable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2423372446
##########
python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py:
##########
@@ -179,8 +179,21 @@ def import_array_stream(
) -> typing.Any:
return polars.from_arrow(handle)
- def import_schema(self, handle: _lib.ArrowSchemaHandle) -> typing.Any:
- raise _lib.NotSupportedError("Polars does not support __arrow_c_schema__")
+ def import_schema(self, handle: _lib.ArrowSchemaHandle) -> polars.Schema:
+ def parse_version(version: str) -> tuple[int, ...]:
+ return tuple(int(v) for v in version.split("."))
+
+ # The version that Polars added support to initialize a schema via the
+ # __arrow_c_schema__ interface
+ required_version = "1.32.2"
+ polars_version = polars.__version__
+ if parse_version(polars_version) >= parse_version(required_version):
+ return polars.Schema(handle)
Review Comment:
Alternatively, could leave the version check and just `return polars.Schema(handle)` immediately.
Would raise with the below before the required version. IMO providing the user a nicer error message is preferable.
```
Traceback (most recent call last):
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/try_import_schema.py", line 11, in <module>
schema = conn.adbc_get_table_schema("test_table_schema")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/dbapi.py", line 521, in adbc_get_table_schema
return self._backend.import_schema(handle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py", line 183, in import_schema
return polars.Schema(handle)
^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/.cache/uv/archive-v0/SC8MnmcDLgKYq5uEoGc4z/lib/python3.11/site-packages/polars/schema.py", line 102, in __init__
for name, tp in input:
TypeError: 'adbc_driver_manager._lib.ArrowSchemaHandle' object is not iterable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2423375875
##########
python/adbc_driver_manager/adbc_driver_manager/_lib.pyi:
##########
@@ -145,16 +145,21 @@ class ArrowArrayHandle:
address: int
is_valid: bool
def release(self) -> None: ...
+ def __arrow_c_array__(
+ self, requested_schema: object | None
+ ) -> tuple[object, object]: ...
class ArrowArrayStreamHandle:
address: int
is_valid: bool
def release(self) -> None: ...
+ def __arrow_c_stream__(self, requested_schema: object | None) -> object: ...
class ArrowSchemaHandle:
address: int
is_valid: bool
def release(self) -> None: ...
+ def __arrow_c_schema__(self) -> object: ...
Review Comment:
This makes the type checker not complain about `polars.Schema(handle)`.
Figured it would be worth adding the other `__arrow_c_...` methods to the stubs too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#issuecomment-3395577637
@henryharbeck it appears polars broke their Python docs? https://docs.pola.rs/api/python/stable/reference/objects.inv needs to exist (and actually index polars.Schema) for the API crosslinking to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#issuecomment-3425343067
Seems like the crosslinking is an existing issue https://github.com/pola-rs/polars/issues/7027
And looks it is not linking correctly in the current docs either (e.g., [fetch_polars](https://arrow.apache.org/adbc/current/python/api/adbc_driver_manager.html#adbc_driver_manager.dbapi.Cursor.fetch_polars)).
I was able to find the (an?) objects.inv file here - https://docs.pola.rs/api/python/stable/objects.inv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2423374715
##########
python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py:
##########
@@ -179,8 +179,19 @@ def import_array_stream(
) -> typing.Any:
return polars.from_arrow(handle)
- def import_schema(self, handle: _lib.ArrowSchemaHandle) -> typing.Any:
- raise _lib.NotSupportedError("Polars does not support __arrow_c_schema__")
+ def import_schema(self, handle: _lib.ArrowSchemaHandle) -> polars.Schema:
+ # The version that Polars added support to initialize a schema via the
+ # __arrow_c_schema__ interface
+ required_version = (1, 32, 2)
+ polars_version = polars.__version__
+ if tuple(int(v) for v in polars_version.split(".")) >= required_version:
+ return polars.Schema(handle)
Review Comment:
Alternatively, could not do the version check and just `return polars.Schema(handle)` immediately.
Would raise with the below before the required version. IMO providing the user a nicer error message is preferable though.
```
Traceback (most recent call last):
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/try_import_schema.py", line 11, in <module>
schema = conn.adbc_get_table_schema("test_table_schema")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/dbapi.py", line 521, in adbc_get_table_schema
return self._backend.import_schema(handle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/development/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/_dbapi_backend.py", line 183, in import_schema
return polars.Schema(handle)
^^^^^^^^^^^^^^^^^^^^^
File "/home/henry/.cache/uv/archive-v0/SC8MnmcDLgKYq5uEoGc4z/lib/python3.11/site-packages/polars/schema.py", line 102, in __init__
for name, tp in input:
TypeError: 'adbc_driver_manager._lib.ArrowSchemaHandle' object is not iterable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "henryharbeck (via GitHub)" <gi...@apache.org>.
henryharbeck commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2447226600
##########
python/adbc_driver_manager/adbc_driver_manager/dbapi.py:
##########
Review Comment:
Yes, I agree it will be inconvenient. FWIW, I _think_ the user manually type hinting the variable assignment works too.
> Else we could template Cursor on the backend type and make the return type dependent on the backend type's associated type.
I will give something a crack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/adbc_driver_manager): Implement `import_schema` for Polars DBAPI backend [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3556:
URL: https://github.com/apache/arrow-adbc/pull/3556#discussion_r2423377394
##########
python/adbc_driver_manager/adbc_driver_manager/dbapi.py:
##########
Review Comment:
Hmm, I think this works but would be inconvenient for people (I'm guessing this will require type assertions all over the place). But I'm not sure of a better way to do this for now. (IIRC associated types are unsupported? Else we could template Cursor on the backend type and make the return type dependent on the backend type's associated type.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org