You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Sunny-Island (via GitHub)" <gi...@apache.org> on 2026/10/05 12:49:24 UTC
Re: [I] Dataset.to_arrow() can generate invalid tables [arrow]
Sunny-Island commented on issue #39670:
URL: https://github.com/apache/arrow/issues/39670#issuecomment-5994777218
I looked into the scan path and think `BasicFragmentEvolution::DevolveSelection` may be a useful place to prevent this inconsistent result. It already identifies fields that are missing from a fragment, and `EvolveBatch` later fills those fields with nulls. Could we reject a missing field there when the Dataset schema declares it non-nullable? This would prevent the scanner from producing a table whose data contradicts its schema, without requiring every fragment to be inspected when the Dataset is created.
I realize this would make the scan fail rather than automatically widen the field to nullable, as suggested in the issue. Since the Dataset and projected schemas are already established by scan time, widening may need to happen earlier during discovery. I would be interested in the maintainers' view on which behavior is preferable, particularly for an explicitly supplied schema versus an inferred one.
This would address the Dataset scan path; the proposed behavior of `pa.unify_schemas()` may need separate consideration.
If the maintainers agree that this behavior should be fixed, I'd be happy to work on a PR. I'd appreciate guidance on whether the preferred behavior is to reject the scan or to widen the inferred schema.
@pitrou, since you discussed the related nullability validation issues in #31387 and #41667, would you have a view on the preferred behavior here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org