You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2024/01/18 15:18:44 UTC
[I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
syun64 opened a new issue, #278:
URL: https://github.com/apache/iceberg-python/issues/278
### Feature Request / Improvement
I see three ways a user would want to create an Iceberg table:
1. Completely manual - by specifying the schema, field by field
2. By inferring the schema from an existing strongly-typed file or pyarrow table
3. By copying the schema of an existing iceberg table (migration)
create_table function currently takes a pyiceberg.Schema as the input. The existing visitors support patterns (1) and (3), but not (2).
This is because the creation of a pyiceberg.Schema is only supported in the following two ways:
1. From a pyarrow schema with valid field-id metadata
2. using a NameMapping which have field-ids. Currently, the only way to create a NameMapping is by constructing it field-id by field-id, or by using a utility function on an existing Iceberg Schema.
Therefore, we need to update an existing Visitor, or create a new Visitor in order to support the generation of a pyiceberg.Schema from a pyarrow Schema with no IDs.
On https://github.com/apache/iceberg-python/pull/219 the following approaches have been discussed so far:
1. Update _ConvertToIceberg to create a pyiceberg.Schema from pyarrow schema by assigning "-1" field_ids and use _SetFreshIDs to assign ordered fresh IDs. This idea unfortunately does not work as _SetFreshIDs requires different IDs to track each column and assign new ones.
2. Create a new Visitor _CreateMappingFromPyArrowSchema that creates a name mapping from PyArrow schema and assigns fresh IDs if it does not have one. This is different from existing _CreateMapping visitor which is a pyiceberg Schema visitor.
3. Use a separate visitor _ConvertToIcebergWithFreshIds which assigns fresh IDs based on the order of the fields' appearance in the pyarrow schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1910270303
@syun64 Yes that sounds like a reasonable proposal to me. On thing to mention. We would also like to use PyIceberg without Arrow, and we can do this by making the type annotation lazy:
```
def create_table(
self,
identifier: Union[str, Identifier],
schema: Union[Schema, "pa.Schema"],
location: Optional[str] = None,
partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
sort_order: SortOrder = UNSORTED_SORT_ORDER,
properties: Properties = EMPTY_DICT,
) -> Table:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1902557930
Thanks for summarizing the approaches and explanation on the concerns.
> I’m not convinced that we can assign ids without relying on the position when generating the name mapping.
Same question as @syun64. My understanding is the `_CreateMappingFromPyArrowSchma` will be very similar to `assign_fresh_schema_ids` which incremently assigns new ids by position as we visit the given schema.
I have another related question. If we
1. create a name-mapping from `pa.Table`
2. use the name-mapping to generate a new iceberg schema
3. use the new iceberg schema to create the table.
what do we do with the name-mapping created in step 1 after the table is created? Do we just discard it or put it in `schema.name-mapping.default`? If the later, I think we need either to update the [`new_table_metadata`](https://github.com/apache/iceberg-python/blob/83306104a25a4ecd1f2185ec46cd9fda247544f4/pyiceberg/table/metadata.py#L399-L409) to not assign fresh ids when a name-mapping present or update the `_SetFreshIds` to respect name-mapping if given. I would appreciate any thoughts on this matter!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900574539
That sounds good @Fokko
I think having a _CreateMappingFromPyArrowSchma preorder visitor does a good job of separating out the two concerns above.
> I think the outcome will be the same as the pre-order visitor, but we don’t do it by position, but by name.
I think this bit about not doing it by position is catching me a bit off guard because I’m not convinced that we can assign ids without relying on the position when generating the name mapping. Just to make sure we are on the same page, this new Visitor will:
1. Map field_ids from PyArrow Schema if the field_id exists
2. Have a Boolean flag to _assign_fresh_ids by ignoring existing field_ids (or an automatic fallback to assign ids if field_ids don’t exist) and assign field ids **by position**
And then, we will use the name mapping generated from the pyarrow schema to assign field ids **by name** and create a new Iceberg Schema.
Does that approach sound consistent with your current thought?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1910272795
> @syun64 Yes that sounds like a reasonable proposal to me. On thing to mention. We would also like to use PyIceberg without Arrow, and we can do this by making the type annotation lazy:
>
> ```
> def create_table(
> self,
> identifier: Union[str, Identifier],
> schema: Union[Schema, "pa.Schema"],
> location: Optional[str] = None,
> partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
> sort_order: SortOrder = UNSORTED_SORT_ORDER,
> properties: Properties = EMPTY_DICT,
> ) -> Table:
> ```
Oh! That's very neat. Thank you for the suggestion :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-2791154801
@0x26res Thanks for reaching out here. All clients re-assign the IDs to make sure that they are unique. The idea is that users don't have to worry about setting the IDs. This has been coming up more often, so maybe we should also allow skipping the re-assignment of IDs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 closed issue #278: Create Iceberg Table from pyarrow Schema with no IDs
URL: https://github.com/apache/iceberg-python/issues/278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-2790415821
Sorry I'm not sure if this is the right place to ask this question.
My understanding from this conversation is that when a user provides a `pa.Schema` to create an iceberg table, field_ids gets "refreshed" (aka, we assign monotonically increasing field_id).
But it seems to me that actually, field_id are refreshed no matter what one does, and in particular when the use provide explicit values for field_ids:
```pyhon
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
NestedField,
StringType,
TimestampType,
)
def test_id_refreshed():
schema = Schema(
NestedField(
field_id=1, name="datetime", field_type=TimestampType(), required=True
),
NestedField(field_id=3, name="symbol", field_type=StringType(), required=True),
)
catalog = InMemoryCatalog("test")
catalog.create_namespace("test_namespace")
table = catalog.create_table(
identifier="test_namespace.bids",
schema=schema,
)
# IDs have been refreshed:
assert table.schema() == Schema(
NestedField(
field_id=1, name="datetime", field_type=TimestampType(), required=True
),
# The field_id should be 3
NestedField(field_id=2, name="symbol", field_type=StringType(), required=True),
)
```
Is that the intention? How can one control the field_id within its schema?
For context I'm trying to have the field_id match the number from my protobuf schema (which I use as the source of truth for my schema).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900264983
Thank you for adding the context @Fokko :)
Just so that we make sure new readers aren’t confused, do you think it’s fair to say that we are talking about ‘assigning fresh IDs’ in two separate ways?
1. Auto assigning fresh IDs in column order when reading iceberg table files
2. Assigning fresh IDs for the purpose of new Iceberg table creation based on an arrow table that does not have field_ids
The discussions so far, and the PR above, have alluded to the fact that (1) above is dangerous and will not be supported in Python and Java. (2) however is necessary for applications to use PyIceberg in a similar way that Spark is being used. Spark Dataframes currently can be used to create new iceberg tables without field_id metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-2791789643
> [@0x26res](https://github.com/0x26res) Thanks for reaching out here. All clients re-assign the IDs to make sure that they are unique. The idea is that users don't have to worry about setting the IDs. This has been coming up more often, so maybe we should also allow skipping the re-assignment of IDs.
@Fokko actually I probably shouldn't assign the IDs myself.
I think iceberg requires requires all ID to be unique including for nested fields. Which protobuf doesn't require. So Iceberg Field IDs and protobuf field number are not compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1906620668
Alright, I went to the source and talked with @danielcweeks and @rdblue. It looks like we made things more complicated than actually needed.
So when reading and writing Parquet, we need to make sure that the IDs are aligned properly. When we are working with runtime data (`pa.Table`'s) then we match everything up based on names.
I also discussed with Dan about adding Arrow types to the `create_table` statement, and he liked the idea, where I was a bit reluctant. But thinking of it, I think it makes sense since it will allow us to create Iceberg tables from a dataframe:
```python
catalog = load_catalog()
catalog.create_table('some.table', df=df)
```
And then:
```python
# It will wire up the schema by name
tbl.overwrite(df)
```
```python
# Should be quite easy with union by name:
tbl.append(df, merge_schema=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1906647358
That makes sense @Fokko .
Just to make sure we are on the same page, does the following approach align with your thoughts?
We are proposing to update the create_table API to:
```
def create_table(
self,
identifier: Union[str, Identifier],
schema: Union[Schema, pa.sSchema],
location: Optional[str] = None,
partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
sort_order: SortOrder = UNSORTED_SORT_ORDER,
properties: Properties = EMPTY_DICT,
) -> Table:
...
if isinstance(schema, pa.Schema):
schema = pre_order_visit_pyarrow(schema, _ConvertToIcebergWithFreshIds())
...
# existing code
```
We will call the function like:
```
table: pa.Table
catalog = load_catalog()
catalog.create_table('some.table', schema=table.schema)
```
And use the previously proposed Visitor: https://github.com/syun64/iceberg-python/blob/preorder-fresh-schema/pyiceberg/io/pyarrow.py#L994 since [new_table_metadata](https://github.com/apache/iceberg-python/blob/46b25be424d1ef0f28778b0873c4d91bf117a2a7/pyiceberg/table/metadata.py#L399) has to take `field_id`ed Iceberg Schema as the input?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1899892752
To add some more context. As also mentioned in the earlier conversation, I don't think assigning fresh IDs is safe: https://github.com/apache/iceberg-python/pull/219#discussion_r1457510803
On the Java side, this is also being deprecated: https://github.com/apache/iceberg/pull/9324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1900388501
What do you think of the following approach:
> I think the problem here is that we don't have an API like in Spark where we can [nicely hide things](https://github.com/apache/iceberg/blob/main/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L225-L242). I'm tempted to allow creating a table from a PyArrow table create_table_from_table(df: pa.Table), but that mixes in PyArrow into the main API, but refrains us from exposing these things to the user (which isn't super user friendly in general).
This would require:
- We can add this to the Catalog, where we can generalize this for all the catalogs.
- Can create a name mapping from a `pa.Table`, then we can use this name-mapping to assign new IDs on it. I think the outcome will be the same as the pre-order visitor, but we don't do it by position, but by name. We'll reuse the name-mapping that you already added 🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "anupam-saini (via GitHub)" <gi...@apache.org>.
anupam-saini commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1904757960
Hello, I would like to put up a PR as per the discussion above if no one has started working already. Please let me know if this is fine. Also, @syun64 and I work together hence I can get up to speed quickly with the discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Create Iceberg Table from pyarrow Schema with no IDs [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #278:
URL: https://github.com/apache/iceberg-python/issues/278#issuecomment-1904289015
> what do we do with the name-mapping created in step 1 after the table is created? Do we just discard it or put it in schema.name-mapping.default? If the later, I think we need either to update the [new_table_metadata](https://github.com/apache/iceberg-python/blob/83306104a25a4ecd1f2185ec46cd9fda247544f4/pyiceberg/table/metadata.py#L399-L409) to not assign fresh ids when a name-mapping present or update the _SetFreshIds to respect name-mapping if given. I would appreciate any thoughts on this matter!
Great question @HonahX my understanding is that the act of putting in a name mapping into schema.name-mapping.default isn't done automatically by any operation, and requires the user to actually insert the name mapping json as a table property into the iceberg table.
I think regardless of whether we create this visitor to create a name mapping (which in turn will be used to create an iceberg schema), or an iceberg schema directly, it will need have to have the ability to incrementally assign a new id by position. Because we are trying to create a new iceberg schema based on an arrow schema that does not have the field_id metadata.
Imagine we are trying to grab a 100 column parquet file from a vendor and create an Iceberg table based on it, and it doens't have PARQUET:FIELD_ID metadata on its columns. Currently, there's no way to create this iceberg table and ingest this data without manually coding and labelling each and every column using the Iceberg schema types to create an Iceberg schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org