You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "max-parke-scale (via GitHub)" <gi...@apache.org> on 2026/08/29 00:23:02 UTC

[I] [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553) [arrow]

max-parke-scale opened a new issue, #51029:
URL: https://github.com/apache/arrow/issues/51029

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Casting a map array still hard-aborts the process (SIGABRT, not a raised exception) with `Map array keys array should have no nulls` when the keys child carries a validity bitmap — even an all-valid one. This is a residual case of #38553: the fix in #41957 changed `MapArray::ValidateChildData` to use `MayHaveNulls()`, which resolved the no-bitmap repro on that issue, but `MayHaveNulls()` answers true whenever a validity bitmap is present and the null count is unknown — it never counts. A cast's output starts with an unknown null count, so the fatal `ARROW_CHECK_OK(ValidateChildData(...))` in `MapArray::SetData` (`array_nested.cc:912` as of current main) kills the process.
   
   Three-line reproduction, all public API, on pyarrow 24.0.0 (the current release):
   
   ```python
   import pyarrow as pa
   m = pa.array([{"a": "1"}, {"b": "2"}], type=pa.map_(pa.string(), pa.string()))
   m.filter(pa.array([False, True])).cast(pa.map_(pa.large_string(), pa.large_string()))
   ```
   
   ```
   /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/array_nested.cc:912:  Check failed: _s.ok() Operation failed: ValidateChildData(data->child_data)
   Bad status: Invalid: Map array keys array should have no nulls
   ```
   
   Exit code 134. The `filter` call itself succeeds and the filtered array passes `validate(full=True)`.
   
   `filter` matters only because it materializes new child arrays that carry an all-valid validity bitmap; `slice` re-views the original buffers without one, which is why the slice→cast variant works after #41957. The bitmap alone is sufficient — same abort with no filter involved:
   
   ```python
   import pyarrow as pa
   keys = pa.array(["a", "b"])
   validity = pa.array([True, True]).buffers()[1]  # all-valid bitmap
   keys = pa.Array.from_buffers(pa.string(), 2, [validity, keys.buffers()[1], keys.buffers()[2]])
   m = pa.MapArray.from_arrays(pa.array([0, 1, 2], type=pa.int32()), keys, pa.array(["1", "2"]))
   m.cast(pa.map_(pa.large_string(), pa.large_string()))  # aborts
   ```
   
   Forcing null-count computation on the input (`m.keys.null_count` → 0) does not help: the cast produces a fresh keys `ArrayData` whose count is unknown again.
   
   Real-world impact: pyiceberg's scan path does filter-then-cast (row-filtered Parquet read, then cast to large types), so any row-filtered scan that projects a map column kills the worker process.
   
   Two candidate fixes, possibly both:
   
   - `MapArray::ValidateChildData` (`array_nested.cc:899`/`905`) could force-compute the null count (`GetNullCount() != 0`) rather than ask `MayHaveNulls()`; counting an all-valid bitmap is cheap next to aborting the process.
   - The `ARROW_CHECK_OK` at `array_nested.cc:912` turns an `Invalid` status into process death even when the data is genuinely invalid; a returned/raised error would be recoverable.
   
   Environment: pyarrow 24.0.0 (PyPI wheel), Python 3.13.13, macOS 26.4.1 arm64.
   
   ### Component(s)
   
   C++, Python
   
   🤖 — posted via [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #51029: [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553)
URL: https://github.com/apache/arrow/issues/51029


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553) [arrow]

Posted by "AnuragRaut08 (via GitHub)" <gi...@apache.org>.
AnuragRaut08 commented on issue #51029:
URL: https://github.com/apache/arrow/issues/51029#issuecomment-5461062328

   I'd like to take this issue up and work on the fix. I'll first reproduce the crash and investigate the MapArray validation/casting path, then add a regression test covering the all-valid validity bitmap case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553) [arrow]

Posted by "AnuragRaut08 (via GitHub)" <gi...@apache.org>.
AnuragRaut08 commented on issue #51029:
URL: https://github.com/apache/arrow/issues/51029#issuecomment-5474597425

   Implemented the fix in PR #51093.
   The `MapArray` validation now correctly handles all-valid validity bitmaps with an unknown null count, and the regression tests pass successfully.
   Please have a look when you get a chance.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Python] Casting a MapArray whose keys child carries an all-valid validity bitmap aborts the process (residual case of GH-38553) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #51029:
URL: https://github.com/apache/arrow/issues/51029#issuecomment-5510345925

   Issue resolved by pull request 51093
   https://github.com/apache/arrow/pull/51093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org