You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/07 08:49:09 UTC
[PR] fix(io): keep nulls when pushing NotEqualTo down to Arrow [iceberg-python]
jackylee-ch opened a new pull request, #3918:
URL: https://github.com/apache/iceberg-python/pull/3918
# Rationale for this change
Pushing `NotEqualTo` down to Arrow drops every row where the column is null: an Arrow comparison yields null for a null input, and `filter` treats that as no match.
```python
table = pa.table({"foo": [None, "hello", "world"]})
table.filter(pc.field("foo") != "hello") # ["world"] -- the null is gone
```
Every other evaluator treats a null as satisfying `NotEqualTo`. `_ExpressionEvaluationVisitor.visit_not_equal` returns `None != value`; `_StrictMetricsEvaluationVisitor.visit_not_equal` reports `ROWS_MUST_MATCH` for a column proven to hold only nulls; and the Arrow `NOT IN` path already keeps them, because `~isin(...)` is false for a null. The reference implementation agrees: `Evaluator.notEq` is `!eq`, and `eq` compares with a nulls-first comparator, so a null never equals the literal.
Keep the null explicitly. `visit_not_in` needs no change.
## Are these changes tested?
Yes, `test_expr_not_equal_to_pyarrow_keeps_nulls` asserts that the pushed-down filter and the in-memory evaluator return the same rows. It and the existing `test_expr_not_equal_to_pyarrow` both fail without the change.
## Are there any user-facing changes?
A `row_filter` using `!=`, or a `NOT IN` that reduces to a single literal, now returns rows where the column is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: NotEqualTo push-down drops null rows (draft, see description) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3918:
URL: https://github.com/apache/iceberg-python/pull/3918#issuecomment-6049673088
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(io): keep nulls when pushing NotEqualTo down to Arrow [iceberg-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3918:
URL: https://github.com/apache/iceberg-python/pull/3918#discussion_r3948136590
##########
pyiceberg/io/pyarrow.py:
##########
@@ -919,7 +919,10 @@ def visit_equal(self, term: BoundTerm, literal: Literal[Any]) -> pc.Expression:
return pc.field(self._get_field_name(term)) == _convert_scalar(literal.value, term.ref().field.field_type)
def visit_not_equal(self, term: BoundTerm, literal: Literal[Any]) -> pc.Expression:
- return pc.field(self._get_field_name(term)) != _convert_scalar(literal.value, term.ref().field.field_type)
+ # A null is not equal to the literal, but an Arrow comparison yields null for it and
+ # the row would be dropped. Keep it explicitly to match the other evaluators.
+ ref = pc.field(self._get_field_name(term))
+ return ref.is_null(nan_is_null=False) | (ref != _convert_scalar(literal.value, term.ref().field.field_type))
Review Comment:
`visit_not_equal` now explicitly treats nulls as matching (via `is_null(...) or (ref != ...)`). This changes the semantics of a delete/overwrite filter like `col != X`: null rows should now be considered *matching* the filter (and therefore be removed/replaced). However, `_expression_to_complementary_pyarrow` currently preserves nulls for predicates that the `_NullNaNUnmentionedTermsCollector` considers "null-unmentioned" (including `BoundNotEqualTo`), which will cause the complementary/preserve filter used during rewrite deletes/overwrites to incorrectly keep null rows for `!=` filters.
To keep delete/overwrite behavior consistent with the new `NotEqualTo` semantics, update `_NullNaNUnmentionedTermsCollector.visit_not_equal` (and likely `visit_not_in` for the same reason) so these predicates are treated as explicitly handling nulls (and NaNs if applicable), preventing `_expression_to_complementary_pyarrow` from OR-ing `is_null`/`is_nan` back into the preserve filter. Adding a focused unit/integration test for overwrite/delete with a nullable column and a `!=` row_filter would help prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: NotEqualTo push-down drops null rows (draft, see description) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3918:
URL: https://github.com/apache/iceberg-python/pull/3918#issuecomment-6205190444
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: NotEqualTo push-down drops null rows (draft, see description) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3918: PyArrow: NotEqualTo push-down drops null rows (draft, see description)
URL: https://github.com/apache/iceberg-python/pull/3918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org