You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2024/01/18 22:14:51 UTC
[I] REPLACE TABLE Support [iceberg-python]
syun64 opened a new issue, #281:
URL: https://github.com/apache/iceberg-python/issues/281
### Feature Request / Improvement
SparkCatalog supports "REPLACE TABLE ... AS SELECT" as an atomic operation when using a SparkCatalog.
Similarly, we should introduce support for an atomic RTAS operation in PyIceberg.
Atomic table replacement creates a new snapshot with the results of the SELECT query, but keeps table history.
https://iceberg.apache.org/docs/latest/spark-ddl/#replace-table--as-select
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-3374713612
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901252873
Makes sense @Fokko . Thank you very much for taking the time to lay all these options out for the cases where a user may have to handle schema updates.
As you suggested, I think it would make sense to port over the Java code to support the replace operation.
For most cases where there are compatible schema changes, I think we would want to rely on **union_by_name_with** function to evolve the schema.
Things definitely get a little hairy when we are handling backwards incompatible type changes, and I think the update_schema(allow_incompatible_changes=True) case and REPLACE TABLE has similar, but slightly different outcomes:
1. update_schema(allow_incompatible_changes=True) as the name suggests allows for schema updates with incompatible changes. This means that updates to specific fields with incompatible changes are allowed. The output list of fields that represent the table are still a union of the existing schema and the new schema
2. buildReplacement from the java code assigns fresh IDs on the new schema, but also ensures that field IDs of field names that were used before will keep their old field IDs. This is also an incompatible schema update, and it is more destructive in the sense that it could delete existing columns (if they are not in the new schema) within the same transaction as well.
I think depending on the agreement that the data provider has with their consumers, both types of incompatible schema changes are valid use cases (albeit awful to apply).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #281: REPLACE TABLE Support
URL: https://github.com/apache/iceberg-python/issues/281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "nicor88 (via GitHub)" <gi...@apache.org>.
nicor88 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900185062
> The next step would be to automatically evolve the schema as well (with flags for compatible/incompatible changes).
@Fokko evolving schema would be nice, also would it be possible to evolve partitions? e.g. after a specific overwrite I want to have schema evolution and possibility to replace the table adding partitions to specific columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-4210033552
Hey all! I do think this feature request has value as discussed in this thread, could we remove staleness here?
Similar to how https://github.com/apache/iceberg-python/pull/498 supports CTAS, I'd propose that the replace functionality be achieved via `replace_table` and `replace_table_transaction` APIs (then changes to the resultant transaction fulfils RTAS workflows). I threw together a draft, https://github.com/apache/iceberg-python/pull/3220, as what that would look like (currently just for the REST catalog), would be interested in knowing the community's thoughts on the direction!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1904323323
@syun64 I started on #284 today. It re-uses the `UpdateSchema` API which already sets the right IDs, and maintains IDs for the existing fields. I don't think that doesn't help us because the `UnionByName` visitor requires an Iceberg schema, instead of a PyArrow schema.
My suggestion is to create a name-mapping from an Arrow table, that we can use in `pyarrow_to_schema` to correctly assign fields by name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1905187291
Hi @Fokko - sounds like you beat me to it 😄 Please let me know if you need any additional heavy lifting on #284 . Happy to help as always.
The reason I was curious if there's an opportunity to deduplicate code here, is because [buildReplacement](https://github.com/apache/iceberg/blob/e32df0ce08086758c44e9174c582638068244073/core/src/main/java/org/apache/iceberg/TableMetadata.java#L672) code in Java also takes Iceberg Schema as an input. It then compares the new updated schema against the existing schema to use the existing ID if the corresponding field name exists, or [assign a new ID](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/types/AssignFreshIds.java#L51) starting from the next increment from [last_column_id](https://github.com/apache/iceberg-python/blob/acfa5bc4e1e0ab167b7d3652438288e9d295f9b6/pyiceberg/table/metadata.py#L153) of the table.
On second thought, I'm wondering if it would actually make sense to extend functionality of the PyArrow Schema Visitor we are planning to implement for https://github.com/apache/iceberg-python/issues/278 and have the schema visitor take the last_column_id and the base Schema as the input so that we assign the existing field ID if it exists, and assign a new field ID that starts from last_column_id. What are your thoughts on this idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-2367035445
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1971024124
@syun64 Yes, that sounds like a great suggestion to me. I think it makes a lot of sense to create a table including a snapshot. I'm not sure if `as_select` is the best name since we just pass in a dataframe.
For most catalogs, this will be quite straightforward. For the REST catalog, there is one subtle implementation detail that we want to take into account, and that we need to leverage the [staged creation](https://github.com/apache/iceberg/blob/bb53c3d4e0e27ac6706803c2371793ad2476ae04/open-api/rest-catalog-open-api.yaml#L2796-L2797). Details can be found here: https://github.com/apache/iceberg/blob/bb53c3d4e0e27ac6706803c2371793ad2476ae04/open-api/rest-catalog-open-api.yaml#L480-L492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1904310699
In order to reduce duplication of code, would it make sense to combine the job of [TypeUtil.assignFreshIds](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/types/TypeUtil.java#L232) with [UnionByNameVisitor](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java)?
They seem to be doing the same task of ensuring that the new schema reuses the field_id of a column that existed in the original schema. The only difference is that TypeUtil.assignFreshIds drops columns that are not in the target schema, where as UnionByNameVisitor unions the original and target schemas.
If that sounds like a good idea, https://github.com/apache/iceberg-python/issues/284 will be a prerequisite to building this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1959949307
There's a PR in progress that will introduce 'REPLACE TABLE' support, but I don't think we've come to a consensus yet on how we would want to support 'AS SELECT' semantics in PyIceberg, and if we want to introduce it at all.
If we wanted to, I think we could update all `create_table...` and `replace_table...` operations that take something like
`as_select: pa.Table = None`
as an optional parameter and add a snapshot update with full table static overwrite to the new table metadata. Does that sound like a reasonable enhancement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-2745930662
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901289940
Makes sense @Fokko . Thank you very much for taking the time to lay all these options where a user may have to handle compatible or incompatible schema updates.
As you suggested, I think it would make sense to port over the Java code to support the replace operation for the case where we just want to replace the existing table with the new schema and data in a single transaction. I’ll read through the code and take a stab at the PR that will introduce the operation to PyIceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900076148
@syun64 thanks for raising this. With the freshly merged write support, we can do:
```python
cat = load_catalog('default')
tbl = cat.load_table('default.some_table')
tbl.overwrite(df: pa.Table)
```
The overwrite will create a new snapshot with the existing schema. This would only replace the data, and keep the metadata. Would that work for you? The next step would be to automatically evolve the schema as well (with flags for compatible/incompatible changes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901165986
> Evolving schema would be nice, also would it be possible to evolve partitions? e.g. after a specific overwrite I want to have schema evolution and possibility to replace the table adding partitions to specific columns.
Schema evolution is [already possible today](https://py.iceberg.apache.org/api/#schema-evolution). What's missing there is the `unionByName` method that will promote the schema to a new schema:
```python
with table.update_schema() as update:
update.union_by_name_with(new_schema)
```
This will promote the schema to the new schema if compatible. If you also want to allow incompatible changes, you can do:
```python
with table.update_schema(allow_incompatible_changes=True) as update:
update.union_by_name_with(new_schema)
```
By making this explicit, you make sure that you don't break the table for the downstream consumers. If you want to add partition columns in a single transaction, you can do:
```python
with table.transaction() as tx:
with tx.update_schema() as update:
update.union_by_name_with(new_schema)
with tx.update_spec() as update:
update.add_field("dt_transaction", "transaction", DayTransform())
```
Note: The updating partitions is still underway (https://github.com/apache/iceberg-python/pull/245/), but is expected to be included in 0.6.0
> I'm wondering if it would be better to have a separate function that achieves these goals in a single transaction?
We could do something similar as `CREATE OR REPLACE TABLE` as in Spark. It isn't just a DROP/CREATE, since it will create a new schema with new field-ids and such. Please refer to the [java code](https://github.com/apache/iceberg/blob/e32df0ce08086758c44e9174c582638068244073/core/src/main/java/org/apache/iceberg/TableMetadata.java#L672). This might be more applicable if you just want to replace everything, instead of evolving the existing table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900741649
Thank you for the great points @Fokko and @nicor88 .
Just like @nicor88 mentioned, I think RTAS will be slightly different from overwrite in the sense that the schema, the partitioning scheme, sort order or any of the table properties can also be updated atomically with this operation. In short, the function needs to support updating any of the arguments that are currently supported on [create_table](https://github.com/apache/iceberg-python/blob/94d7821cbc6b31b791e18d4f91c0991684616076/pyiceberg/catalog/__init__.py#L286) function, in addition to overwriting the Iceberg table data with the input pyarrow table.
I'm wondering if it would be better to have a separate function that achieves these goals in a single transaction?
```
class Table:
...
def replace(
self,
schema: Schema,
df: pa.Table,
location: Optional[str] = None,
partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
sort_order: SortOrder = UNSORTED_SORT_ORDER,
properties: Properties = EMPTY_DICT,
) -> None:
# update table properties, partition spec, sort_order and schema
# overwrite all data in the table with new data from df
# commit transaction in single metadata update
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] REPLACE TABLE Support [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #281:
URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-3321964113
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org