You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CaselIT (via GitHub)" <gi...@apache.org> on 2025/10/27 12:51:26 UTC
[I] Postgresql driver does not support multidimensional arrays [arrow-adbc]
CaselIT opened a new issue, #3621:
URL: https://github.com/apache/arrow-adbc/issues/3621
### What happened?
The driver does not seem to support multidimensional arrays:
- Reading multi dimensional array literal flattens them:
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("SELECT '{{a,b},{b,c}}'::text[][] as x")
table = cur.fetch_arrow_table()
print(table)
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}
```
- Same happens when reading from a table:
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE text_table (text_data text[], text_data2 text[][])""")
cur.execute("INSERT INTO text_table VALUES ('{x,y}', '{{a,b},{c,d}}')")
cur.execute("SELECT text_data, text_data2 FROM text_table")
res = cur.fetchall()
print(res)
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
```
- insert data into multidimensional array does not work
```py
from adbc_driver_postgresql.dbapi import connect
url = 'postgresql://scott:tiger@localhost/test'
with connect(url) as conn:
with conn.cursor() as cur:
cur.execute("""CREATE TABLE int_table (int_data int[], int_data2 int[][])""")
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
```
Similar erros happen with text types. Passing arrow.scalars does not help either, regardless of splicit typs. Same error with either of these lines:
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ]), pa.scalar([[1,2], [2,3]])))`
- `cur.execute("INSERT INTO int_table VALUES ($1, $2)", (pa.scalar([1,2 ], pa.list_(pa.int32())), pa.scalar([[1,2], [2,3]], pa.list_(pa.list_(pa.int32())))))`
have not tried updates, but I guess a similar error would happen.
For my use care the most problematic issue is the fact that dimensionality is lost while reading multidimensional data without any error being reported
### Stack Trace
```
Traceback (most recent call last):
assert table.to_pydict() == {'x': [[['a', 'b'], ['b', 'c']]]}, table.to_pydict()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'x': [['a', 'b', 'b', 'c']]}
```
```
Traceback (most recent call last):
assert res == [(["x", "y"], [["a", "b"], ["c", "d"]])], res
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [(['x', 'y'], ['a', 'b', 'c', 'd'])]
```
```
Traceback (most recent call last):
File "", line 279, in <module>
cur.execute("INSERT INTO int_table VALUES ($1, $2)", ([1,2 ], [[1,2], [2,3]]))
File "C:\Users\csl\miniconda3\envs\mamba\Lib\site-packages\adbc_driver_manager\dbapi.py", line 751, in execute
handle, self._rowcount = _blocking_call(
^^^^^^^^^^^^^^^
File "adbc_driver_manager/_lib.pyx", line 1770, in adbc_driver_manager._lib._blocking_call
return func(*args, **kwargs)
File "adbc_driver_manager/_lib.pyx", line 1364, in adbc_driver_manager._lib.AdbcStatement.execute_query
check_error(status, &c_error)
File "adbc_driver_manager/_lib.pyx", line 261, in adbc_driver_manager._lib.check_error
raise convert_error(status, error)
adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1016 not found
```
trying with text raises a similar error:
> adbc_driver_manager.InternalError: INTERNAL: nanoarrow call failed: PostgresType::FromSchema(type_resolver, bind_schema->children[i], &type, &na_error) = (22) Invalid argument. Postgres array type with child oid 1009 not found
### How can we reproduce the bug?
See examples above.
### Environment/Setup
adbc_driver_manager = 1.8.0
adbc_driver_postgresql = 1.8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Postgresql driver does not support multidimensional arrays [arrow-adbc]
Posted by "CaselIT (via GitHub)" <gi...@apache.org>.
CaselIT commented on issue #3621:
URL: https://github.com/apache/arrow-adbc/issues/3621#issuecomment-3457385203
understood, makes sense considering the architecture
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Postgresql driver does not support multidimensional arrays [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #3621:
URL: https://github.com/apache/arrow-adbc/issues/3621#issuecomment-3455924182
The data has to be representable in Arrow. (The driver itself knows nothing about Python; it is a generic driver and adbc-driver-manager is bridging it with Python.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Postgresql driver does not support multidimensional arrays [arrow-adbc]
Posted by "CaselIT (via GitHub)" <gi...@apache.org>.
CaselIT commented on issue #3621:
URL: https://github.com/apache/arrow-adbc/issues/3621#issuecomment-3455332747
> IIRC, Postgres multidimensional arrays can have a different set of dimensions in every row, right?
that seems to be the case. I think postgresql just has an array type, it doesn't care about the number of dimensions. This script passes correctly
```sql
create table t(
x int[],
y int[][]
);
insert into t values ('{1,2}', '{{1},{2}}'), ('{{1},{2}}', '{1,2}');
select * from t
```
returns
```
x |y |
---------+---------+
{1,2} |{{1},{2}}|
{{1},{2}}|{1,2} |
```
From an user point of view I would have preferred an error here, with possibly an option on connection (or similar) called something like `flatted_multidimensional_arrays` to enable this behaviour
regarding suggested arrow type to return, I'm not an expert here, so not sure.
If an option similar to the above is added you could decide to bail out depending on the option value and return a series of python objects if the is no more appropriate type in arrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Postgresql driver does not support multidimensional arrays [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #3621:
URL: https://github.com/apache/arrow-adbc/issues/3621#issuecomment-3453855167
I think we explicitly punted on this at the time given that Arrow didn't have any real type for this. IIRC, Postgres multidimensional arrays can have a different set of dimensions in every row, right? Now that we have [fixed_size_tensor](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor), we could try mapping to this type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org