You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "bigluck (via GitHub)" <gi...@apache.org> on 2025/03/03 15:17:09 UTC
[I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
bigluck opened a new issue, #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567
### What happened?
With the latest version of the library (1.4.0) this query:
```sql
SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname
```
fails with:
```
adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR: no binary output function available for type aclitem
```
I can't control the input SQL query, but I could "disable", if it's possible, the binary mode for all the queries against the `pg_catalog` namespace, but it isn't supported.
### Stack Trace
```
[postgres ] 2025-03-03 15:06:30.901 GMT [56969] ERROR: no binary output function available for type aclitem
[postgres ] 2025-03-03 15:06:30.901 GMT [56969] STATEMENT: SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname
[postgres ] 2025-03-03 15:06:30.907 GMT [56969] WARNING: there is no transaction in progress
Traceback (most recent call last):
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 105, in receive
return self.receive_nowait()
^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 100, in receive_nowait
raise WouldBlock
anyio.WouldBlock
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 118, in receive
return receiver.item
^^^^^^^^^^^^^
AttributeError: 'MemoryObjectItemReceiver' object has no attribute 'item'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 157, in call_next
message = await recv_stream.receive()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 120, in receive
raise EndOfStream
anyio.EndOfStream
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bigluck/dev/src/main.py", line 104, in catch_exceptions_middleware
out = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
with collapse_excgroups():
File "/nix/store/icv5w5d2sl202gxpz98hl33hglkir5c0-python3-3.12.3/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/slowapi/middleware.py", line 136, in dispatch
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
await responder(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
await self.app(scope, receive, self.send_with_gzip)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
await self.app(scope, otel_receive, otel_send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/src/router.py", line 86, in custom_route_handler
out: Response = await original_route_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bigluck/dev/src/routes/route_create_query.py", line 304, in route_create_query
raise e
File "/Users/bigluck/dev/src/routes/route_create_query.py", line 296, in route_create_query
cur.execute(res_query)
File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 698, in execute
handle, self._rowcount = _blocking_call(
^^^^^^^^^^^^^^^
File "adbc_driver_manager/_lib.pyx", line 1546, in adbc_driver_manager._lib._blocking_call_impl
File "adbc_driver_manager/_lib.pyx", line 1217, in adbc_driver_manager._lib.AdbcStatement.execute_query
File "adbc_driver_manager/_lib.pyx", line 260, in adbc_driver_manager._lib.check_error
adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR: no binary output function available for type aclitem
. SQLSTATE: 42883
INFO: 127.0.0.1:59923 - "POST /v0/bi/query HTTP/1.1" 500 Internal Server Error
```
### How can we reproduce the bug?
```python
with (
adbc_driver_postgresql.dbapi.connect(CONNECTION_URI) as conn,
conn.cursor() as cur,
):
# https://github.com/apache/arrow-adbc/issues/2093
cur.adbc_statement.set_options(**{'adbc.postgresql.use_copy': 'false'})
cur.execute("SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname")
```
### Environment/Setup
Postgresql 15.5
Python 3.12.3
adbc-driver-postgresql 1.4.0
pyarrow 19.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697210439
Ah, so they really do need to be a single driver/connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2702485278
That sounds reasonable to me (in that it's not too much additional effort, and should hopefully unblock this case?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697066856
For the fetching, is the Arrow support important or would just plain ol' psycopg work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2695819211
https://www.postgresql.org/message-id/CAAvxfHdpbCjJ6VO%2BBmJgMVA1yu50xBAEGw2NWaNnug-W0rBHCg%40mail.gmail.com
> I think it's at least somewhat intentional, to have a core type
> that has no binary I/O so that that case can be tested.
That's rather annoying. Hmm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697003632
Yeah, I found it too.
My case is not a regular one. I'm not using the library to import/export data into a PostgreSQL database but to create empty PG tables starting from a list of arrow schemas (it's a weird case, I know) and to fetch some PG-related results (like the query above).
So I'm more than happy if I can turn off the binary mode support used by the library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2698025795
Could we do a "text only" version where the output schema is always `struct<string, string, string, ...>` and just return the raw libpq text output? (The text output is how we issue our own calls to the catalog and everything is in theory using the same helper under the hood with the binary/text flag turned on/off).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697302228
Unfortunately there's not a way to "turn off" the binary support since the _only_ support is the binary support (we never built a text format parser!) So the only way to support this would be to build a separate set of text format parsers from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697195620
Ideally, I would prefer to retrieve the results as an Arrow table. However, I am open to alternative solutions. The core challenge I face is that I need to execute a query within a temporary context immediately after creating tables and schemas, followed by a rollback to the original state. Essentially, I'm building an ephemeral execution context specifically for the input query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697269079
yes, unfortunately I have no other options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org