You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mariotaddeucci (via GitHub)" <gi...@apache.org> on 2025/02/26 04:50:30 UTC
[I] [Feature] - Allow Schema Overwrite [iceberg-python]
mariotaddeucci opened a new issue, #1726:
URL: https://github.com/apache/iceberg-python/issues/1726
### Feature Request / Improvement
It would be beneficial to introduce an option in pyiceberg to completely overwrite the schema as part of the table update process. By allowing users to fully replace the schema through a simple API, we can leverage the existing schema evolution mechanism to offer a more complete solution for schema management, specifically when overwrite a table.
Proposed Changes:
- Introduce a new method that allows users to replace the entire table schema.
I am very interested in working on this feature if it is approved. Please let me know your thoughts and any potential guidelines or additional requirements for this contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1726: [Feature] - Allow Schema Overwrite
URL: https://github.com/apache/iceberg-python/issues/1726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "tabrezm (via GitHub)" <gi...@apache.org>.
tabrezm commented on issue #1726:
URL: https://github.com/apache/iceberg-python/issues/1726#issuecomment-3155484950
> Can you share more about the context around this functionality?
Adding my $.02 here ... I was looking for a direct replacement for `createOrReplace` like we have in [Spark](https://iceberg.apache.org/docs/1.5.0/spark-writes/#creating-tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1726:
URL: https://github.com/apache/iceberg-python/issues/1726#issuecomment-2692854196
Hey @mariotaddeucci Can you share more about the context around this functionality? Also, based on what I see in the implementation in #1727 I'm reluctant adding this.
In the PR, if you have a table with data, and you only replace the schema using the newly added API, it will return rows with all null values (since they are projected into the schema). If you want an API similar to [Union by Name](https://py.iceberg.apache.org/api/#union-by-name), where you update the fields based on the changes, then this makes sense to me. For example:
```python
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, DoubleType
catalog = load_catalog()
initial_schema = Schema(
NestedField(1, "city_name", StringType(), required=False),
NestedField(2, "latitude", FloatType(), required=False),
NestedField(3, "longitude", FloatType(), required=False),
)
table = catalog.create_table("default.locations", initial_schema)
new_schema = Schema(
NestedField(1, "city", StringType(), required=False),
NestedField(2, "lat", DoubleType(), required=False),
NestedField(3, "long", DoubleType(), required=False),
NestedField(4, "population", LongType(), required=False),
)
with table.update_schema() as update:
update.overwrite(new_schema)
# The big difference here is that if the field ID already exists, then we update the name/type/doc etc
assert new_schema == tbl.schema()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1726:
URL: https://github.com/apache/iceberg-python/issues/1726#issuecomment-3911167878
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on issue #1726:
URL: https://github.com/apache/iceberg-python/issues/1726#issuecomment-2684966894
+1 - I'd also like for schema replacement to be supported like this, unless I'm mistaken and it is already.
Without https://github.com/apache/iceberg-python/pull/433, a transactional "schema-replace-then-table-overwrite" gets us close to non-cumbersome (i.e. as opposed to individually changing columns) table replacement (but there are still differences vs Spark of course, like the snapshot log not being cleared).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Feature] - Allow Schema Overwrite [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1726:
URL: https://github.com/apache/iceberg-python/issues/1726#issuecomment-3832369590
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org