You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "bigluck (via GitHub)" <gi...@apache.org> on 2025/03/03 15:17:09 UTC

[I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

bigluck opened a new issue, #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567

   ### What happened?
   
   With the latest version of the library (1.4.0) this query:
   
   ```sql
   SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname
   ```
   
   fails with:
   ```
   adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR:  no binary output function available for type aclitem
   ```
   
   I can't control the input SQL query, but I could "disable", if it's possible, the binary mode for all the queries against the `pg_catalog` namespace, but it isn't supported.
   
   ### Stack Trace
   
   ```
   [postgres   ] 2025-03-03 15:06:30.901 GMT [56969] ERROR:  no binary output function available for type aclitem
   [postgres   ] 2025-03-03 15:06:30.901 GMT [56969] STATEMENT:  SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname
   [postgres   ] 2025-03-03 15:06:30.907 GMT [56969] WARNING:  there is no transaction in progress
   
   Traceback (most recent call last):
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 105, in receive
       return self.receive_nowait()
              ^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 100, in receive_nowait
       raise WouldBlock
   anyio.WouldBlock
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 118, in receive
       return receiver.item
              ^^^^^^^^^^^^^
   AttributeError: 'MemoryObjectItemReceiver' object has no attribute 'item'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 157, in call_next
       message = await recv_stream.receive()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 120, in receive
       raise EndOfStream
   anyio.EndOfStream
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/Users/bigluck/dev/src/main.py", line 104, in catch_exceptions_middleware
       out = await call_next(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
       raise app_exc
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
       await self.app(scope, receive_or_disconnect, send_no_error)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
       with collapse_excgroups():
     File "/nix/store/icv5w5d2sl202gxpz98hl33hglkir5c0-python3-3.12.3/lib/python3.12/contextlib.py", line 158, in __exit__
       self.gen.throw(value)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
       raise exc
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
       response = await self.dispatch_func(request, call_next)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/slowapi/middleware.py", line 136, in dispatch
       response = await call_next(request)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
       raise app_exc
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
       await self.app(scope, receive_or_disconnect, send_no_error)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
       await responder(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
       await self.app(scope, receive, self.send_with_gzip)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
       await self.app(scope, otel_receive, otel_send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
       await self.app(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
       await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
       raise exc
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
       await app(scope, receive, sender)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
       await self.middleware_stack(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
       await route.handle(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
       await self.app(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
       await wrap_app_handling_exceptions(app, request)(scope, receive, send)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
       raise exc
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
       await app(scope, receive, sender)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
       response = await f(request)
                  ^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/src/router.py", line 86, in custom_route_handler
       out: Response = await original_route_handler(request)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
       raw_response = await run_endpoint_function(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
       return await run_in_threadpool(dependant.call, **values)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
       return await anyio.to_thread.run_sync(func, *args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
       return await get_async_backend().run_sync_in_worker_thread(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
       return await future
              ^^^^^^^^^^^^
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
       result = context.run(func, *args)
                ^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/bigluck/dev/src/routes/route_create_query.py", line 304, in route_create_query
       raise e
     File "/Users/bigluck/dev/src/routes/route_create_query.py", line 296, in route_create_query
       cur.execute(res_query)
     File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 698, in execute
       handle, self._rowcount = _blocking_call(
                                ^^^^^^^^^^^^^^^
     File "adbc_driver_manager/_lib.pyx", line 1546, in adbc_driver_manager._lib._blocking_call_impl
     File "adbc_driver_manager/_lib.pyx", line 1217, in adbc_driver_manager._lib.AdbcStatement.execute_query
     File "adbc_driver_manager/_lib.pyx", line 260, in adbc_driver_manager._lib.check_error
   adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR:  no binary output function available for type aclitem
   . SQLSTATE: 42883
   
   INFO:     127.0.0.1:59923 - "POST /v0/bi/query HTTP/1.1" 500 Internal Server Error
   ```
   
   ### How can we reproduce the bug?
   
   ```python
   with (
       adbc_driver_postgresql.dbapi.connect(CONNECTION_URI) as conn,
       conn.cursor() as cur,
   ):
       # https://github.com/apache/arrow-adbc/issues/2093
       cur.adbc_statement.set_options(**{'adbc.postgresql.use_copy': 'false'})
       cur.execute("SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname")
   ```
   
   ### Environment/Setup
   
   Postgresql 15.5
   Python 3.12.3
   adbc-driver-postgresql 1.4.0
   pyarrow 19.0.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697210439

   Ah, so they really do need to be a single driver/connection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2702485278

   That sounds reasonable to me (in that it's not too much additional effort, and should hopefully unblock this case?).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697066856

   For the fetching, is the Arrow support important or would just plain ol' psycopg work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2695819211

   https://www.postgresql.org/message-id/CAAvxfHdpbCjJ6VO%2BBmJgMVA1yu50xBAEGw2NWaNnug-W0rBHCg%40mail.gmail.com
   
   > I think it's at least somewhat intentional, to have a core type
   > that has no binary I/O so that that case can be tested.
   
   That's rather annoying. Hmm. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697003632

   Yeah, I found it too.
   My case is not a regular one. I'm not using the library to import/export data into a PostgreSQL database but to create empty PG tables starting from a list of arrow schemas (it's a weird case, I know) and to fetch some PG-related results (like the query above).
   So I'm more than happy if I can turn off the binary mode support used by the library.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2698025795

   Could we do a "text only" version where the output schema is always `struct<string, string, string, ...>` and just return the raw libpq text output? (The text output is how we issue our own calls to the catalog and everything is in theory using the same helper under the hood with the binary/text flag turned on/off).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697302228

   Unfortunately there's not a way to "turn off" the binary support since the _only_ support is the binary support (we never built a text format parser!) So the only way to support this would be to build a separate set of text format parsers from scratch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697195620

   Ideally, I would prefer to retrieve the results as an Arrow table. However, I am open to alternative solutions. The core challenge I face is that I need to execute a query within a temporary context immediately after creating tables and schemas, followed by a rollback to the original state. Essentially, I'm building an ephemeral execution context specifically for the input query.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/postgres: no binary output function available for type aclitem [arrow-adbc]

Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #2567:
URL: https://github.com/apache/arrow-adbc/issues/2567#issuecomment-2697269079

   yes, unfortunately I have no other options


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org