You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sungwy (via GitHub)" <gi...@apache.org> on 2024/11/01 13:07:32 UTC
[I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
sungwy opened a new issue, #1284:
URL: https://github.com/apache/iceberg-python/issues/1284
### Feature Request / Improvement
Currently, `create_table` takes a `pyiceberg.schema.Schema` or `pyarrow.Schema`. The API ignores the `field_id`s provided in the schema and issues a set of new ones.
Not only does this cause confusion for the users, because `pyiceberg.schema.Schema` requires `field_id`s, but it also means that a table cannot be created with the guarantee that its schema will match the field IDs of the provided schema. This prevents the API from being able to be used for **table migrations** ([discussion on example use case](https://github.com/apache/iceberg-python/issues/1227#issuecomment-2414821679)), where a user would want to take the following course of steps:
1. `catalog.load_table` to load the `pyiceberg.table.Table` of an existing Iceberg table.
2. get `pyiceberg.schema.Schema` of the loaded table
3. create a new table in the target catalog using `catalog.create_table` using the existing table's schema
4. copy over files into the new table using `add_files` (this is not possible yet, but there is a discussion that would allow `add_files` to work on a table with schema with matching field_ids)
Above procedure will not work, unless we introduce an enhancement to `create_table` that enables creation of a new table that matches the field_ids of the provided Schema.
One way of addressing this issue will be to have two ways of representing the Iceberg table's Schema.
1. If a schema without field_id is passed into the API, we should create a new `pyiceberg.schema.Schema` with newly assigned field_ids to create the table.
2. If a schema with field_id is passed into the API, we should use the exact same `pyiceberg.schema.Schema` to create the table.
I discuss a few ideas in achieving this below, all with their own pros and cons:
### Create a subclass of `pyiceberg.schema.Schema` without `field_id`
This sounds like the best approach, but once explored, we quickly realize that this may be impossible. The main challenge with this approach is the fact that `pyiceberg.schema.Schema` describes its fields using `NestedField` which are nested structures of pydantic BaseModels with `field_id` as a required field. So there isn't a way to create a subclass class of `pyiceberg.schema.Schema` without `field_id`.
### Create a variant class of `pyiceberg.schema.Schema` without `field_id`
This is a bit different from above approach, and requires us to make variant classes of `pyiceberg.schema.Schema`, that are not subclassed from it. This is not ideal, because we will have to maintain field_id-less copies of [NestedField](https://github.com/apache/iceberg-python/blob/d559e53ed1895f947274c23de754b802a3f6c46f/pyiceberg/types.py#L282-L305), [StructType](https://github.com/apache/iceberg-python/blob/d559e53ed1895f947274c23de754b802a3f6c46f/pyiceberg/types.py#L350-L362), [MapType](https://github.com/apache/iceberg-python/blob/d559e53ed1895f947274c23de754b802a3f6c46f/pyiceberg/types.py#L465-L477), [ListType](https://github.com/apache/iceberg-python/blob/d559e53ed1895f947274c23de754b802a3f6c46f/pyiceberg/types.py#L415C7-L424) and Schema and create methods to create a field_id'ed Schema from its field_id-less variant. It is possible, but it will be hard and messy to manage.
### We could make `field_id` an optional attribute of `NestedField` and field_id'd iceberg Types
This will allow us to create `pyiceberg.schema.Schema` with and without field_ids. However, this creates a new opportunity for issues to be introduced into PyIceberg, that have been prevented with the NestedField's attributes directly matching that of the REST Catalog spec. With `field_id` as an optional field, we will need to introduce a lot more validations across our code base to ensure that the field_id is set in all the nested fields within a schema before using it.
### Keep the `field_id`s when using `pyiceberg.schema.Schema`, but generate new ones when using `pyarrow.Schema`
I think this may be the safest approach. This will be user-friendly, `pyiceberg.schema.Schema` requires `field_id`s, and `pyarrow.Schema` does not. `pyarrow.Schema` is also just a completely different class, so users do not hold the expectation that the `field_id`s within a `pyarrow.Schema` are kept in the `pyiceberg.schema.Schema` (although this is an enhancement we could introduce in the future). When and if we introduce new Schema representations as alternate input schema representations to the API, we could evaluate whether it would make sense to keep the field IDs or assign new ones case by case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2469667659
> In general, I think the philosophy should be; that people don't have to worry about field IDs, and this should be hidden away.
+1
> In the case of the table migration, that's some advanced situation where you don't want to re-assign the IDs
We can have a function to explicitly assign IDs, such as `assign_fresh_schema_ids`.
If Iceberg Schema is given, it will use its original assigned ID. If PyArrow Schemas is given, it will be converted to Iceberg Schemas and have IDs assigned.
@sungwy WDYT of this method instead of adding the `assign_fresh_ids` flag to the functions? I think it doesn't make sense for the `create_table` function to have a flag for schema ID assignment.
I guess the difference is, is there a case where I want to call `create_table` with Iceberg Schema and have `assign_fresh_ids=False`? Can the user just call `assign_fresh_schema_ids(iceberg_schema)` instead?
We don't care about PyArrow Schemas since they are always converted and assigned IDs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2910804096
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-4227463123
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2452399942
Thanks for the writeup!
>Keep the field_ids when using pyiceberg.schema.Schema, but generate new ones when using pyarrow.Schema
+1 to this
Here's my take:
Currently, `create_table` can take in either a Pyiceberg Schema class (`pyiceberg.schema.Schema`) or PyArrow Schema class (`pyarrow.Schema`).
Inside `create_table`, PyArrow Schema is converted to PyIceberg Schema. And then the field_ids are all reassigned.
https://github.com/apache/iceberg-python/blob/d559e53ed1895f947274c23de754b802a3f6c46f/pyiceberg/catalog/rest.py#L573-L576
There are 3 ways a user can interact with the `create_table` API:
* Passing in a PyIceberg Schema that already has field_ids assigned
* Passing in a PyIceberg Schema that needs field_ids assignment
* Passing in a PyArrow Schema
For the first case, we should not reassign the field_ids, but we do today.
The the second case, we should assign field_ids as it is required.
The the third case, we should convert to PyIceberg Schema and assign field_ids.
I think it'll be more user-friendly for the `create_table` function to accept either
1. PyIceberg Schema with field_ids already assigned
2. PyArrow Schema, in which case, we'll do the conversion and field_ids assignment
Today, I think the only way to craft a PyIceberg Schema that needs field_ids assignment is either through creating it by hand or using the `_convert_schema_if_needed` function.
We might need a way to check the validity of the field_ids in case the Pyiceberg Schema was created by hand.
And we can always couple `_convert_schema_if_needed` with `assign_fresh_schema_ids` to ensure that the PyIceberg Schemas always have a valid field_id assignment.
I'm excited about this change. This might also help us streamline `create_table` with partition_spec and sort_order since there's been an issue with the field_id mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "talawahtech (via GitHub)" <gi...@apache.org>.
talawahtech commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-6049731593
Bumping so this doesn't get closed. I've noticed a lot of issues that I follow being marked as stale/auto-closed recently. Should I just comment when that happens if the problem still exists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2454746766
Thanks for bringing this up!
> I'm excited about this change. This might also help us streamline create_table with partition_spec and sort_order since there's been an issue with the field_id mismatch.
Indeed :) To provide some historical context. The idea was that PyIceberg was more of a library than an end-user tool, therefore it was always a second layer (like having PyArrow in front of it).
In general, I think the philosophy should be; that people don't have to worry about field IDs, and this should be hidden away. In the case of the table migration, that's some advanced situation where you don't want to re-assign the IDs. I can think of two options:
- Having a flag to not assign fresh IDs
- Ability to have a custom assigning class, so you can also do other ways of assigning the IDs (pre-order, post-order etc).
Keep in mind that the REST catalog also might re-assign IDs (I think the one we use in the integration tests also does this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2469668954
looks like in `new_table_metadata` the ID assignment is propagated to the `partition_spec` and `sort_order` as well
https://github.com/apache/iceberg-python/pull/1304/files#diff-a7fbd8c6b0564308ca872ba479f0e240ff2c893d15776a14e52b91b48e9ac470R524-R528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2957353324
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1284: Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema
URL: https://github.com/apache/iceberg-python/issues/1284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "talawahtech (via GitHub)" <gi...@apache.org>.
talawahtech commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-3393934835
I just ran into this issue as well and it was extremely confusing. It was even more complicated from me because I was trying to use required fields with a Dataframe coming from Polars (where nullable=True by default). I was hitting validation errors because of the nullable differences, so I tried to cast my polars Dataframe using my predefined Iceberg schema and it failed because of filed_id mismatches.
```
arrow_data = polars_df.to_arrow().cast(my_predefined_iceberg_schema.as_arrow())
iceberg_table.append(arrow_data)
```
My understanding of the recommended approach is:
1) If you are not using partitions or sorting, then you can use `create_table` with a pyarrow schema so you don't have to think about IDs.
```
schema = pa.schema([
pa.field("foo", pa.timestamp(), nullable=False),
pa.field("bar", pa.int32(), nullable=True),
])
catalog.create_table(identifier="docs_example.bids", schema=schema)
```
2) If you want to use partitions, use the `create_table_transaction` method instead
```
with catalog.create_table_transaction(identifier="docs_example.bids", schema=schema) as txn:
with txn.update_spec() as update_spec:
update_spec.add_field("foo", transform=DayTransform())
update_spec.add_identity("bar")
```
3) If you want to use partitions **and** sorting, wait for the [new `update_sort_order`](https://github.com/apache/iceberg-python/pull/2552) function to make it into a release so you can use it with `create_table_transaction`.
Unfortunately, `create_table_transaction` does not work with certain REST catalog implementations like AWS S3 Tables, because they [don't currently support `stage-create`](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html#endpoint-considerations). So none of the options above work for me.
My current "workaround is to use `create_table` with `pyiceberg.schema.Schema` and `PartitionSpec` and `SortOrder`, but **add big warning** to my code to remind myself that I shouldn't use that schema for anything else because the **field_id values are not honored**. For example, if you need to cast your data to match the schema of the table then use `catalog.load_table(table_id).schema()` to retrieve the REAL schema.
It would be great if there were a cleaner way to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-6049669467
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2503087394
Thanks for the context yesterday, I was still noodling on it overnight.
If I understand correctly (and please also share the video of you and @adrianqin; I must have missed it during the paternity leave), you're looking for a shallow clone of a table. In this situation, the metadata and manifests are recreated, but the re-use the existing data-files are re-used to avoid unnecessary heavy lifting. As I mentioned yesterday, this is still a bit tricky since you might do a delete operation, where a data-file is being dropped that's still referenced in the other table, but that's inherent to a shallow clone.
Instead of mangling the `create_table` operation, shouldn't it be better to have a clone operation?
> The case where some Iceberg Rest Catalog Servers do not respect the given field IDs and assigns fresh IDs actually feels more like an edge case to me, that we want to opine on and course correct.
The REST catalog follows the philosophy that clients shouldn't have to worry about Field-IDs. The register table is designed to take an existing file, and re-use all the metadata, rather than re-assigning it. But it looks like there is interest in more flavors than just `create` and `register`. A similar question was also raised on Slack: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1732441495465079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2452415487
Hey @kevinjqliu thanks for the support! Glad to hear we are on the same page regarding this issue.
As with all API changes, I think we would want to deliberate on these options carefully, so I'll keep this issue open for a bit longer for others to opine before moving on to implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2465872766
I just put up this PR to introduce a new flag `assign_fresh_ids` on `create_table` and related methods.
> Keep in mind that the REST catalog also might re-assign IDs (I think the one we use in the integration tests also does this).
Yes I think this is extremely great to make note of. I ran into this issue when setting up the integration tests against the REST Catalog image and I was a bit confused. I understand that it is upto the REST Catalog Server to decide how it wants to create the table once the request is accepted, but I can't help but find it very counter-intuitive that the REST Catalog takes id assigned Schema, PartitionSpec and SortOrder and would still assign fresh IDs ☹️
Regardless, I'm excited to introduce this feature, so that we can support more migration workflows through PyIceberg - which is something many users have been hoping to do through Python applications 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enhance `catalog.create_table` API to enable creation of table with matching `field_ids` to provided Schema [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #1284:
URL: https://github.com/apache/iceberg-python/issues/1284#issuecomment-2474284318
> If Iceberg Schema is given, it will use its original assigned ID. If PyArrow Schemas is given, it will be converted to Iceberg Schemas and have IDs assigned.
@kevinjqliu - Yes, I actually feel more inclined to drive this based on the type of the input schema. To me, the fact that the Iceberg Schema is required to have IDs assigned makes a very strong case for us to respect the given field IDs and propagate it to the Iceberg table metadata.
The case where some Iceberg Rest Catalog Servers do not respect the given field IDs and assigns fresh IDs actually feels more like an edge case to me, that we want to opine on and course correct.
Ideally, when the IDs are specified on the Schema, it should be respected by the server. Else, the Schema supplied in a **createTableRequest** should not have field IDs assigned in the request.
This approach aligns with my initial proposal on the issue description ⬆️
I put up the PR with the `assign_fresh_ids` flag based on @Fokko 's concerns with above approach. Could I ask for your reviews on this PR, and maybe we can decide on the approach there? https://github.com/apache/iceberg-python/pull/1304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org