You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "GujaLomsadze (via GitHub)" <gi...@apache.org> on 2026/08/09 14:06:12 UTC

[PR] GH-46918: [Python] Respect explicit Field type in add_column/set_column [arrow]

GujaLomsadze opened a new pull request, #50834:
URL: https://github.com/apache/arrow/pull/50834

   ### Rationale for this change
   
   Closes #46918
   
   `add_column` / `set_column` on both `Table` and `RecordBatch` convert the column
   data before they look at `field_`. Column data that is entirely null is therefore
   inferred as the null type, and `AddColumn` / `SetColumn` then reject it against an
   explicitly typed `Field`:
   
   ```python
   >>> t = pa.table({"A": [1.0, 2.0]})
   >>> t.append_column(pa.field("B", pa.float64(), nullable=True), [[None, None]])
   ArrowInvalid: Field type did not match data type
   ```
   
   The type the user asked for was available in `field_` the whole time, it just was
   not used for the conversion.
   
   ### What changes are included in this PR?
   
   When `field_` is a `Field` and the converted column came out as null type while the
   field type is not null, the column is cast to the field type. Four call sites, same
   two lines each:
   
   - `Table.add_column`
   - `Table.set_column`
   - `RecordBatch.add_column`
   - `RecordBatch.set_column`
   
   `append_column` on both classes forwards to `add_column`, so it is covered too.
   
   The issue only reports `append_column`, but `set_column` has the identical defect,
   so it is fixed here as well. Happy to split it out if reviewers prefer.
   
   Data that is not all-null keeps its inferred type, so a genuine mismatch is still
   reported rather than silently cast:
   
   ```python
   >>> t.append_column(pa.field("B", pa.int64()), [[1.5, 2.5]])
   ArrowInvalid: Field type did not match data type   # unchanged
   ```
   
   This narrow scope is deliberate. Always converting with the field type would be more
   uniform with `pa.array(data, type=...)`, but that API truncates silently
   (`pa.array([1.5, 2.5], type=pa.int64())` gives `[1, 2]`), so the broad version would
   turn today's loud error into a silent data change. Happy to switch to the broad
   behaviour if that is preferred.
   
   ### Are these changes tested?
   
   Yes. `test_table_add_column_all_null_data_typed_field` (parametrized over
   `Table`/`RecordBatch` x `add_column`/`set_column`) and
   `test_table_append_column_all_null_data_typed_field`. They cover the typed field
   case, a null field with null data staying null, and a non-null mismatch still
   raising.
   
   ### Are there any user-facing changes?
   
   Yes, a bug fix. Calls that raised `ArrowInvalid` / `ArrowTypeError` now succeed and
   produce a column of the requested type. No previously working call changes behaviour.
   
   ### Note
   
   One pre-existing inconsistency noticed while writing the tests, left alone here:
   for the same type mismatch `Table` raises `ArrowInvalid` while `RecordBatch` raises
   `ArrowTypeError`. Can file a separate issue if that is worth aligning.
   
   ### AI usage disclosure
   
   This change was written with the assistance of an AI coding agent (Claude Code).
   The agent located the four call sites, wrote the patch and the tests, and ran the
   test suite. I reviewed the diff, chose the narrow-vs-broad scope, and verified the
   results before submitting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-46918: [Python] Respect explicit Field type in add_column/set_column [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50834:
URL: https://github.com/apache/arrow/pull/50834#issuecomment-5231927775

   :warning: GitHub issue #46918 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-46918: [Python] Respect explicit Field type in add_column/set_column [arrow]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #50834:
URL: https://github.com/apache/arrow/pull/50834#issuecomment-5244412738

   LGTM, a straightforward and well-designed fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org