You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2025/02/18 15:17:03 UTC
[I] Push upsert changes detection to Arrow [iceberg-python]
Fokko opened a new issue, #1679:
URL: https://github.com/apache/iceberg-python/issues/1679
### Feature Request / Improvement
Right now we iterate [over all the rows](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/upsert_util.py) to find the rows that are different. Only these rows are updated, which is great. However, Arrow is probably faster since it pushes it down to C++. Therefore I came up with the following:
```python
def get_rows_to_update(source_table: pa.Table, target_table: pa.Table, join_cols: list[str]) -> pa.Table:
"""
Return a table with rows that need to be updated in the target table based on the join columns.
When a row is matched, an additional scan is done to evaluate the non-key columns to detect if an actual change has occurred.
Only matched rows that have an actual change to a non-key column value will be returned in the final output.
"""
all_columns = set(source_table.column_names)
join_cols_set = set(join_cols)
non_key_cols = all_columns - join_cols_set
diff_expr = functools.reduce(operator.or_, [pc.field(f"{col}-lhs") != pc.field(f"{col}-rhs") for col in non_key_cols])
return (source_table
.join(target_table, keys=list(join_cols_set), join_type='inner', left_suffix='-lhs', right_suffix='-rhs')
.filter(diff_expr)
.drop_columns([f"{col}-rhs" for col in non_key_cols])
.rename_columns({f"{col}-lhs" if col not in join_cols else col: col for col in source_table.column_names})
)
```
Unfortunately, it looks like Arrow doesn't carry through the nullability of the column, resulting in an incompatible schema. Raised an issue on the Arrow side here https://github.com/apache/arrow/issues/45557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "mattmartin14 (via GitHub)" <gi...@apache.org>.
mattmartin14 commented on issue #1679:
URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2679973969
@Fokko - good stuff here :). I see you all are starting to ramp up the efforts with working on the upsert stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1679:
URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2692857608
@kevinjqliu Yes, let's close this 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on issue #1679:
URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2666511285
As a temporary workaround, would something like this suffice?
```python
def force_non_null_fields(table: pa.Table, fields_to_force: list[str]) -> pa.Table:
new_fields = [
field.with_nullable(False) if field.name in fields_to_force else field
for field in table.schema
]
new_schema = pa.schema(new_fields)
return pa.Table.from_arrays(table.columns, schema=new_schema)
```
My understanding is that this creates a new table with our desired schema without copying the underlying arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1679:
URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2669693184
@corleyma I was incorporating your code, but then I realized that it is even simpler: https://github.com/apache/iceberg-python/pull/1685/files#diff-cdc27639285a3481a96d65f8844d47245b60afd400d5a5da80f83bc0895ce2b2R70
Thanks again for the suggestion, that made me come to the final solution :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1679:
URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2679985290
I think this is done in #1685, right? @Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Push upsert changes detection to Arrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #1679: Push upsert changes detection to Arrow
URL: https://github.com/apache/iceberg-python/issues/1679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org