You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mdwint (via GitHub)" <gi...@apache.org> on 2025/09/04 15:49:31 UTC
[I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
mdwint opened a new issue, #2426:
URL: https://github.com/apache/iceberg-python/issues/2426
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Upserting a table fails when the input dataframe contains `None` in a join column.
I've reproduced this error by editing `test_upsert_with_nulls` from #1861, adding this at the end:
```python
# upsert table with null value
data_with_null = pa.Table.from_pylist(
[
{"foo": None, "bar": 1, "baz": False},
],
schema=schema,
)
upd = table.upsert(data_with_null, join_cols=["foo"])
```
The `foo` column contains `None`, causing `TypeError: Invalid literal value: None`.
```
tests/table/test_upsert.py:720: in test_upsert_with_nulls
upd = table.upsert(data_with_null, join_cols=["foo"])
pyiceberg/table/__init__.py:1343: in upsert
return tx.upsert(
pyiceberg/table/__init__.py:798: in upsert
matched_predicate = upsert_util.create_match_filter(df, join_cols)
pyiceberg/table/upsert_util.py:37: in create_match_filter
return In(join_cols[0], unique_keys[0].to_pylist())
pyiceberg/expressions/__init__.py:682: in __new__
literals_set: Set[Literal[L]] = _to_literal_set(literals)
pyiceberg/expressions/__init__.py:52: in _to_literal_set
return {_to_literal(v) for v in values}
pyiceberg/expressions/__init__.py:52: in <setcomp>
return {_to_literal(v) for v in values}
pyiceberg/expressions/__init__.py:59: in _to_literal
return literal(value)
pyiceberg/expressions/literals.py:159: in literal
raise TypeError(f"Invalid literal value: {repr(value)}")
E TypeError: Invalid literal value: None
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-3254544130
thanks for the report!
i can reproduce it locally. the issue seems to be related to the `foo` column as the `join_cols`.
We hit this branch https://github.com/apache/iceberg-python/blob/52d810efb62e39ec6d8d6a2f4cd2cad8165e2d2c/pyiceberg/table/upsert_util.py#L33-L37
Seems like the `In` expression cannot handle None/Null values
https://github.com/apache/iceberg-python/blob/52d810efb62e39ec6d8d6a2f4cd2cad8165e2d2c/pyiceberg/expressions/__init__.py#L678-L682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "mdwint (via GitHub)" <gi...@apache.org>.
mdwint commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-3257719740
Computing the inner join in Python solved the issue. I've now opened #2429 for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "mdwint (via GitHub)" <gi...@apache.org>.
mdwint commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-3254942924
I will try implementing a fix tomorrow, but I may come back with some questions since I don't know the codebase yet.
I'm guessing `IsNull` would be the correct filter predicate. Does that sound right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-5861302179
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "mdwint (via GitHub)" <gi...@apache.org>.
mdwint commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-3255586794
I've made some progress fixing the filter expressions in https://github.com/apache/iceberg-python/compare/main...mdwint:iceberg-python:fix/upsert-with-nulls
The tests now confirm that the right expressions are produced. For example, in the last test case I can print this:
```
Or(
left=And(
left=IsNull(term=Reference(name='foo')),
right=EqualTo(term=Reference(name='bar'), literal=LongLiteral(1))
),
right=And(
left=EqualTo(term=Reference(name='foo'), literal=literal('lemon')),
right=IsNull(term=Reference(name='bar'))
)
)
```
However, that last test case fails to update the row `{"foo": None, "bar": 1, ...}` with `join_cols=["foo", "bar"]`:
```
tests/table/test_upsert.py:741: in test_upsert_with_nulls
assert upd.rows_updated == 1
E assert 0 == 1
E + where 0 = UpsertResult(rows_updated=0, rows_inserted=1).rows_updated
```
So I think the filter expressions are now correct, but the upsert logic still doesn't handle updates with null values correctly. Wild guess: maybe it has to do with the way null values are compared (null doesn't equal itself?). Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-5657420543
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "mdwint (via GitHub)" <gi...@apache.org>.
mdwint commented on issue #2426:
URL: https://github.com/apache/iceberg-python/issues/2426#issuecomment-3257392284
> So I think the filter expressions are now correct, but the upsert logic still doesn't handle updates with null values correctly. Wild guess: maybe it has to do with the way null values are compared (null doesn't equal itself?).
I've isolated the problem to step 3 in `get_rows_to_update`: https://github.com/apache/iceberg-python/blob/075a966c96fbb113197652df1ccc8997f39bdedf/pyiceberg/table/upsert_util.py#L123-L124
This join ignores `None` values. It returns no matches, but I want it to match `bar: [[1]] foo: [[null]]`:
```python
(Pdb) p source_index
pyarrow.Table
bar: int32
foo: large_string
__source_index: int64
----
bar: [[1,null]]
foo: [[null,"lemon"]]
__source_index: [[0,1]]
(Pdb) p target_index
pyarrow.Table
bar: int32
foo: large_string
__target_index: int64
----
bar: [[1]]
foo: [[null]]
__target_index: [[0]]
(Pdb) p matching_indices
pyarrow.Table
bar: int32
foo: large_string
__source_index: int64
__target_index: int64
----
bar: []
foo: []
__source_index: []
__target_index: []
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Upsert with None values fails on "Invalid literal value: None" [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2426: Upsert with None values fails on "Invalid literal value: None"
URL: https://github.com/apache/iceberg-python/issues/2426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org