You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "okhsunrog (via GitHub)" <gi...@apache.org> on 2026/01/26 13:49:54 UTC

[D] Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement? [arrow-adbc]

GitHub user okhsunrog created a discussion: Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement?

## Context

We have an Arrow Flight SQL server and noticed that when users call `executemany()` with Python data, it fails for column types smaller than Int64 (e.g., Int32, Int16, Int8).

## Observed Behavior

1. Server returns correct `parameter_schema` after `prepare()`:
```python
cursor.adbc_prepare("INSERT INTO my_table(col) VALUES (?)")
param_schema = cursor._stmt.get_parameter_schema()
# Returns: Schema([Field('$1', int32)])
```

2. User calls `executemany()` with Python integers:
```python
cursor.executemany("INSERT INTO my_table(col) VALUES (?)", [(1,), (2,), (3,)])
```

3. PyArrow converts Python `int` to `Int64` by default (in `convert_executemany_parameters`)

4. Server receives `Int64` data but expects `Int32` resulting in an error:
```
column types must match schema types, expected Int32 but found Int64
```

## Workaround

Users can manually create Arrow data with correct types:
```python
cursor.adbc_prepare("INSERT INTO my_table(col) VALUES (?)")
param_schema = pa.Schema._import_from_c_capsule(
    cursor._stmt.get_parameter_schema().__arrow_c_schema__()
)
batch = pa.record_batch([[1, 2, 3]], schema=param_schema)
cursor.executemany("INSERT INTO my_table(col) VALUES (?)", batch)
# Works!
```

## Question

Is this behavior by design? 

Looking at the code, `convert_executemany_parameters()` in `_dbapi_backend.py` doesn't have access to `parameter_schema` and simply infers types from Python values.

Should ADBC:
1. **Current behavior**: User is responsible for providing correctly typed Arrow data
2. **Alternative**: Auto-cast parameters to match `parameter_schema` when available

If (1) is intentional, it might be worth documenting this more explicitly.


GitHub link: https://github.com/apache/arrow-adbc/discussions/3911

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement? [arrow-adbc]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
GitHub user CurtHagenlocher added a comment to the discussion: Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement?

I think (1) is the only realistic answer for the spec as a whole, though individual drivers might be able to do (2) under some circumstances.

GitHub link: https://github.com/apache/arrow-adbc/discussions/3911#discussioncomment-15607372

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org


Re: [D] Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement? [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
GitHub user lidavidm added a comment to the discussion: Should execute() /executemany() auto-cast parameters to match parameter_schema from prepared statement?

Right, the conversion from Python values to Arrow is best-effort. While we could use the schema from GetParameterSchema instead, I'm hesitant since I think for many systems, the schema is not fully reflective of the actual types (e.g. I know Postgres will give VARCHAR when the type is ambiguous/multiple types are possible)

GitHub link: https://github.com/apache/arrow-adbc/discussions/3911#discussioncomment-15612068

----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org