You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "HonahX (via GitHub)" <gi...@apache.org> on 2024/03/05 09:41:37 UTC

[PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

HonahX opened a new pull request, #498:
URL: https://github.com/apache/iceberg-python/pull/498

   This draft PR experiments a possible implementation for `CreateTableTransaction`. At this early stage, I will focus on the high-level requirements and correctness to support staged creation and commit the staged table. The new APIs for `CreateTableTransaction` are subject to change.
   
   Just for your convenience, the current PR:
   On Table Side:
   
   - Add `StagedTable` which represents a uncommitted table, inherits `Table` and disables table scan. 
   - Add `CreateTableTransaction` which inherits `Transaction`, takes in a `StagedTable` and add initial create table updates to the `self._updates`
   - Other helpers to help build metadata from a chain of updates
   
   On Catalog Side:
   
   - Add `_create_staged_table` API which returns a `StagedTable` 
   - Add `create_table_transaction` API 
   - Add a experimental implementation for GlueCatalog
   
   The tentative API looks like
   ```python
   with test_catalog.create_table_transaction(identifier, table_schema_nested) as txn:
           with txn.update_schema() as update_schema:
               update_schema.add_column(path="b", field_type=IntegerType())
   
           txn.set_properties(test_a="test_aa", test_b="test_b", test_c="test_c")
   ```
   
   Appreciate any initial reviews, suggestions and thoughts!
   
   cc @syun64 @Fokko 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542411739


##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also most helper functions in `Catalog` are specific to non-RestCatalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-RestCatalog inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also most helper functions in `Catalog` are specific to non-RestCatalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-Rest Catalogs inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#issuecomment-2009036538

   I updated the implementation to take the third approach:
   
   > a third approach that we may add a flag to TableUpdate to mark it as create changes and let _apply_table_update handle these updates specially. We will exclude this flag from serialization so RestCatalog is not affected. If it can work, we do not need this additional class anymore.
   
   This approach let us get rid of the additional metadata class and requires only small updates on the current `update_table_metadata` mechanism.
   
   I will address other review comments soon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1549588018


##########
pyiceberg/table/__init__.py:
##########
@@ -760,7 +852,10 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     for update in updates:
         new_metadata = _apply_table_update(update, new_metadata, context)
 
-    return new_metadata.model_copy(deep=True)
+    if enforce_validation:
+        return TableMetadataUtil.parse_obj(new_metadata.model_dump())

Review Comment:
   This is probably quite expensive, but I think it is worth paying.



##########
mkdocs/docs/api.md:
##########
@@ -165,6 +165,25 @@ catalog.create_table(
 )
 ```
 
+To create a table with some subsequent changes in a transaction:

Review Comment:
   ```suggestion
   To create a table with some subsequent changes atomically in transaction:
   ```



##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(

Review Comment:
   Thanks @syun64 for breaking the tie here, let me mark this conversation as resolved :)



##########
pyiceberg/catalog/__init__.py:
##########
@@ -717,6 +791,10 @@ def _get_updated_props_and_update_summary(
 
         return properties_update_summary, updated_properties
 
+    @staticmethod
+    def empty_table_metadata() -> TableMetadata:
+        return TableMetadataV1(location="", last_column_id=-1, schema=Schema())

Review Comment:
   Thanks for the detailed explanation, that makes sense to me 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513742498


##########
pyiceberg/catalog/glue.py:
##########
@@ -437,46 +471,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:

Review Comment:
   Actually, now that I look at this, I don't think this is a good idea, since because **TableMetadata** should be required in a **Table**. Although this does leave me a bit confused regarding how [AssertCreate](https://github.com/apache/iceberg-python/blob/14c021be0c7ad7570352c9f17077d67010d40e01/pyiceberg/table/__init__.py#L778) requirement is ever validated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513787586


##########
pyiceberg/catalog/glue.py:
##########
@@ -437,46 +471,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:

Review Comment:
   I think the `AssertCreate` is a safeguard to notify us if the create table commit incorrectly goes into the update table path. i.e. A create table transaction is constructed with an identifier that is already registered in the catalog. In that case, `AssertCreate.validate` raise an exception when validating all the requirements
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1525837679


##########
tests/integration/test_writes.py:
##########
@@ -632,6 +632,60 @@ def test_write_and_evolve(session_catalog: Catalog, format_version: int) -> None
                 snapshot_update.append_data_file(data_file)
 
 
+@pytest.mark.integration
+@pytest.mark.parametrize("format_version", [2])
+def test_create_table_transaction(session_catalog: Catalog, format_version: int) -> None:
+    if format_version == 1:
+        pytest.skip(
+            "There is a bug in the REST catalog (maybe server side) that prevents create and commit a staged version 1 table"

Review Comment:
   My initial guess is that [CatalogHandler](https://github.com/apache/iceberg/blob/f19643a93f5dac99bbdbc9881ef19c89d7bcd3eb/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L336-L342) use the default values to build the initial empty metadata. But the default format-version is now [2](https://github.com/apache/iceberg/blob/6e21bbf4c4cd2c8351a64636f91d05d00492dff2/core/src/main/java/org/apache/iceberg/TableMetadata.java#L911) so it's impossible for the server to build v1 metadata in this case.
   
   I remembered that `iceberg-rest` used the CatalogHandler to wrap the SqlCatalog. I will double-check that this weekend and do more tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1550918541


##########
pyiceberg/table/__init__.py:
##########
@@ -760,7 +852,10 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     for update in updates:
         new_metadata = _apply_table_update(update, new_metadata, context)
 
-    return new_metadata.model_copy(deep=True)
+    if enforce_validation:
+        return TableMetadataUtil.parse_obj(new_metadata.model_dump())

Review Comment:
   👍 Since for create-table-transaction commit, we are generating the entire metadata model via a chain of `model_copy` without any validation, I feel it safer to trigger a validation on the model before return :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513718061


##########
pyiceberg/catalog/glue.py:
##########
@@ -358,6 +365,33 @@ def _get_glue_table(self, database_name: str, table_name: str) -> TableTypeDef:
         except self.glue.exceptions.EntityNotFoundException as e:
             raise NoSuchTableError(f"Table does not exist: {database_name}.{table_name}") from e
 
+    def _create_staged_table(

Review Comment:
   Great suggestion! In the initial implementation I did not pay much attention to the catalog code organization. Let me refactor it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1536944882


##########
pyiceberg/table/metadata.py:
##########
@@ -366,6 +366,11 @@ def construct_partition_specs(cls, data: Dict[str, Any]) -> Dict[str, Any]:
                 fields = data[PARTITION_SPEC]
                 data[PARTITION_SPECS] = [{SPEC_ID: INITIAL_SPEC_ID, FIELDS: fields}]
                 data[DEFAULT_SPEC_ID] = INITIAL_SPEC_ID
+            elif data.get("partition_spec") is not None:

Review Comment:
   This is also included in https://github.com/apache/iceberg-python/pull/544



##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(

Review Comment:
   While I don't have a strong preference, Ithink the name `create_table_transaction` clearly indicates that the API returns a "Transaction." This naming choice becomes particularly relevant when we consider potential usage scenarios. For example:
   ```python
   with catalog.create_table_transaction(
       identifier="docs_example.bids",
       schema=schema,
       location="s3://pyiceberg",
       partition_spec=partition_spec,
       sort_order=sort_order,
   ) as txn:
       with txn.update_schema() as update_schema:
           update_schema.add_column(path="new_column", field_type=StringType())
   
       with txn.update_spec() as update_spec:
           update_spec.add_identity("symbol")
   
       txn.set_properties(test_a="test_aa", test_b="test_b", test_c="test_c")
   ```
   Given this context, I'm inclined to keep the name as it is for now. However, I'm completely open to further discussion and would love to hear any thoughts or suggestions on this matter.
   



##########
pyiceberg/catalog/glue.py:
##########
@@ -435,46 +439,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:
+                requirement.validate(base_metadata)
+
+            updated_metadata = update_table_metadata(base_metadata, table_request.updates)
+            if updated_metadata == base_metadata:
+                # no changes, do nothing
+                return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
+
+            # write new metadata
+            new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
+            new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
+            self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
+
+            update_table_input = _construct_table_input(
+                table_name=table_name,
+                metadata_location=new_metadata_location,
+                properties=current_table.properties,
+                metadata=updated_metadata,
+                glue_table=current_glue_table,
+                prev_metadata_location=current_table.metadata_location,
+            )
 
-        # Pass `version_id` to implement optimistic locking: it ensures updates are rejected if concurrent
-        # modifications occur. See more details at https://iceberg.apache.org/docs/latest/aws/#optimistic-locking
-        self._update_glue_table(
-            database_name=database_name,
-            table_name=table_name,
-            table_input=update_table_input,
-            version_id=glue_table_version_id,
-        )
+            # Pass `version_id` to implement optimistic locking: it ensures updates are rejected if concurrent
+            # modifications occur. See more details at https://iceberg.apache.org/docs/latest/aws/#optimistic-locking
+            self._update_glue_table(
+                database_name=database_name,
+                table_name=table_name,
+                table_input=update_table_input,
+                version_id=glue_table_version_id,
+            )
+
+            return CommitTableResponse(metadata=updated_metadata, metadata_location=new_metadata_location)
+        except NoSuchTableError:
+            # Create the table
+            updated_metadata = construct_table_metadata(table_request.updates)

Review Comment:
   I was thinking of it, and my rationale for the current implementation centers around ensuring a uniform transaction creation and commit process for both RestCatalogs and other types of catalogs. Specifically, for RestCatalogs, it's required to initiate CreateTableTransaction with `_create_table(staged_create=True)` and to use `_commit_table` with both initial and subsequent updates during transaction commitment. On the other hand, alternative catalogs offer more flexibility, allowing for either the use of` _commit_table` to reconstruct table metadata upon commitment or a modified` _create_table` API to create table during the transaction commitment.
   
   Considering pyiceberg's alignment with Rest API principles, where` _commit_table` aggregates metadata updates to construct the revised metadata for table updates within a transaction, it seems prudent to maintain consistency with Rest API practices for table creation within transactions. This approach simplifies the process by relying on `_commit_table` to generate and commit metadata from scratch, eliminating the need to distinguish between RestCatalogs and other catalog types during transaction commitments.
   
   Additionally, I've noted that the existing create_table and new_table_metadata APIs lack support for initializing metadata with snapshot information. I think that responsibility should belong to `AddSnapshotUpdate` and `update_table_metadata`. Thus, I've opted to maintain the current approach of utilizing ` _commit_table` for both functions. 
   
   Does this approach sound reasonable to you? Please feel free to correct me if I've misunderstood any aspect of this process. Thanks for your input!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542411739


##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also many other helper functions in `Catalog` are specific to non-Rest Catalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-Rest Catalogs inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also many other helper functions in `Catalog` are specific to non-Rest Catalogs. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-Rest Catalogs inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542406368


##########
pyiceberg/catalog/__init__.py:
##########
@@ -717,6 +791,10 @@ def _get_updated_props_and_update_summary(
 
         return properties_update_summary, updated_properties
 
+    @staticmethod
+    def empty_table_metadata() -> TableMetadata:
+        return TableMetadataV1(location="", last_column_id=-1, schema=Schema())

Review Comment:
   The default is still V2 table. When creating a createTableTransaction, we first call `new_table_metadata` which by default gives a V2 metadata. Then, we parse the initial metadata to a sequence of `TableUpdates`, including an `UpgradeFormatVersionUpdate`. Finally when we re-building the metadata in `_commit_table`, the `UpgradeFormatVersionUpdate` will bump the metadata to the correct version.
   
   If we use V2Metadata here, we won't be able to create any V1 table since we cannot downgrade from V2 to V1. I think this is the same issue in `iceberg-rest` that prevents us from creating V1 table via create-table-transaction.https://github.com/apache/iceberg-python/pull/498#discussion_r1525837679 I should follow-up that later.
   
   I added some comments to explain the purpose and also make it private. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513218389


##########
pyiceberg/catalog/glue.py:
##########
@@ -358,6 +365,33 @@ def _get_glue_table(self, database_name: str, table_name: str) -> TableTypeDef:
         except self.glue.exceptions.EntityNotFoundException as e:
             raise NoSuchTableError(f"Table does not exist: {database_name}.{table_name}") from e
 
+    def _create_staged_table(

Review Comment:
   I think we might have an opportunity here to move **_create_staged_table** function into [pyiceberg/catalog/__init__.py](https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/__init__.py) and refactor the existing **create_table** functions on the different catalog implementations to all use **_create_staged_table** to create an instance of the StagedTable, and then commit that StagedTable to the catalog backend. 
   
   I think what sets each catalog's implementation of **create_table** apart is how it handles the commit against the catalog backened, but they all seem to share the same sequence of operations in how it instantiates its notion of a new table.
   
   What are your thoughts on this idea @HonahX ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1524685842


##########
pyiceberg/table/__init__.py:
##########
@@ -382,7 +383,54 @@ def commit_transaction(self) -> Table:
             return self._table
 
 
+class CreateTableTransaction(Transaction):
+    @staticmethod
+    def create_changes(table_metadata: TableMetadata) -> Tuple[TableUpdate, ...]:

Review Comment:
   I think we can also accumulate the changes in `_updates` of the `Transaction` itself



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1549581440


##########
mkdocs/docs/api.md:
##########
@@ -165,6 +165,25 @@ catalog.create_table(
 )
 ```
 
+To create a table with some subsequent changes in a transaction:

Review Comment:
   ```suggestion
   To create a table with some subsequent changes atomically in a transaction:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1512522294


##########
pyiceberg/table/__init__.py:
##########
@@ -753,6 +807,143 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     return new_metadata.model_copy(deep=True)
 
 
+class InitialTableMetadataConstructor:

Review Comment:
   I think we need a better way to construct an initial metadata from a chain of updates. The major challenge is that python does not allow an "empty" metadata like [Java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L332-L338).So we cannot directly use `_apply_table_update` or `update_table_metadata` here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513709599


##########
pyiceberg/table/__init__.py:
##########
@@ -753,6 +807,143 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     return new_metadata.model_copy(deep=True)
 
 
+class InitialTableMetadataConstructor:

Review Comment:
   > What are your thoughts on updating the TableMetadataCommonFields Fields to align with the Rest Catalog Open API Spec?
   
   I am also in favor of this approach. I tried to construct an EMPTY_METADATA but failed because of those "before" validators. For example, [construct_partition_specs](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/metadata.py#L328-L345) automatically insert an unpartitioned spec to the metadata if there is no existing partition spec. Let me investigate how to refactor the common fields and these validators to allow a Base TableMetadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513264836


##########
pyiceberg/catalog/glue.py:
##########
@@ -437,46 +471,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:

Review Comment:
   Should we refactor the return values for the **base_metadata** to be Optional so that we can rely on **AssertCreate** requirement validation instead to check if the table already exists when _commit_table is called?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513259769


##########
pyiceberg/table/__init__.py:
##########
@@ -753,6 +807,143 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     return new_metadata.model_copy(deep=True)
 
 
+class InitialTableMetadataConstructor:

Review Comment:
   According to the [Rest Catalog Open API spec](https://github.com/apache/iceberg/blob/1.4.x/open-api/rest-catalog-open-api.yaml#L1570-L1572), only the **format-version** and the **table-uuid** are the only required attributes of the entire Spec. What are your thoughts on updating the TableMetadataCommonFields Fields to align with the Rest Catalog Open API Spec? This will also allow us to create a Base TableMetadata, and use the existing **_apply_table_update** functions instead of having to write a separate constructor.



##########
pyiceberg/table/__init__.py:
##########
@@ -753,6 +807,143 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     return new_metadata.model_copy(deep=True)
 
 
+class InitialTableMetadataConstructor:

Review Comment:
   According to the [Rest Catalog Open API spec](https://github.com/apache/iceberg/blob/1.4.x/open-api/rest-catalog-open-api.yaml#L1570-L1572), only the **format-version** and the **table-uuid** are the only required attributes of the TableMetadata. What are your thoughts on updating the TableMetadataCommonFields Fields to align with the Rest Catalog Open API Spec? This will also allow us to create a Base TableMetadata, and use the existing **_apply_table_update** functions instead of having to write a separate constructor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#issuecomment-2036194320

   Thanks @Fokko @syun64 for detailed reviewing! Since this PR changes/refactors lots of code, I'll merge this first and leave other works (such as supporting createTableTrans in Hive and Sql) to follow-up PRs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#issuecomment-2033344103

   > Shall we move "append", "overwrite", and "add_files" to `Transaction` class? This change would enable us to seamlessly chain these operations with other table updates in a single commit. This adjustment could be particularly beneficial in the context of `CreateTableTransaction`, as it would enable users to not only create a table but also populate it with initial data in one go.
   
   I think this is a great question.
   
   I think we have two options here:
   1. We move these actions into the Transaction class, and remove them from Table class
   2. We move them into the Transaction class, and also keep an implementation in the Table class
   
   I'm not sure which of the above two are better, but I keep asking myself whether there's a 'good' reason why we have two separate APIs that achieve similar results.
   
   For example, we have **update_spec**, **update_schema** that can be created from the **Transaction** or the **Table**, and I feel like we might be creating work for ourselves by duplicating the feature in both classes. What if we consolidated all of our actions into the Transaction class, and removed them from the Table class?
   
   I think the upside of that would be that API would convey a very clear message to the developer that a _transaction is committed to a table_, and that a series of _actions_ can be chained onto the _same transaction_, as a single commit.
   
   In addition, we can avoid [issues like this](https://github.com/apache/iceberg-python/pull/508) where we roll out a feature to one API implementation, but not the other.
   
   ```
       with given_table.update_schema() as tx:
           tx.add_column(path="new_column1", field_type=IntegerType())
   ```
   
   ```
       with given_table.transaction() as tx:
           with tx.update_schema() as update:
               update.add_column(path="new_column1", field_type=IntegerType())
   ```
   
   To me, the bottom pattern feels more explicit than the above option, and I'm curious to hear others' opinions on this topic


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542411739


##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also most other helper functions in `Catalog` are specific to non-RestCatalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-Rest Catalogs inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also most other helper functions in `Catalog` are specific to non-Rest Catalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on naming) and make all non-Rest Catalogs inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1520912688


##########
pyiceberg/table/metadata.py:
##########
@@ -287,6 +287,31 @@ def _generate_snapshot_id() -> int:
     return snapshot_id
 
 
+class IncompleteTableMetadata(TableMetadataCommonFields):

Review Comment:
   With this, we can re-use `_apply_table_update` to construct an initial metadata from a chain of updates



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542406368


##########
pyiceberg/catalog/__init__.py:
##########
@@ -717,6 +791,10 @@ def _get_updated_props_and_update_summary(
 
         return properties_update_summary, updated_properties
 
+    @staticmethod
+    def empty_table_metadata() -> TableMetadata:
+        return TableMetadataV1(location="", last_column_id=-1, schema=Schema())

Review Comment:
   The default is still V2 table. When creating a createTableTransaction, we first call `new_table_metadata` which by default gives a V2 metadata. Then, we parse the initial metadata to a sequence of `TableUpdates`, including an `UpgradeFormatVersionUpdate`. Finally when we re-build the metadata in `_commit_table`, the `UpgradeFormatVersionUpdate` will bump the metadata to the correct version.
   
   If we use V2Metadata here, we won't be able to create any V1 table since we cannot downgrade from V2 to V1. I think this is the same issue in `iceberg-rest` that prevents us from creating V1 table via create-table-transaction.https://github.com/apache/iceberg-python/pull/498#discussion_r1525837679 I should follow-up that later.
   
   I added some comments to explain the purpose and also make it private. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1537251443


##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   The inheritance feels off here. I would just expect to `create_table_transaction`. The current `_create_staged_table` on the `Catalog` is now very specific for non-REST catalogs. Should we introduce another layer in the OOP hierarchy there? Then we can override `create_table_transaction` there. 



##########
pyiceberg/table/__init__.py:
##########
@@ -1707,7 +1831,8 @@ def union_by_name(self, new_schema: Union[Schema, "pa.Schema"]) -> UpdateSchema:
         visit_with_partner(
             Catalog._convert_schema_if_needed(new_schema),
             -1,
-            UnionByNameVisitor(update_schema=self, existing_schema=self._schema, case_sensitive=self._case_sensitive),  # type: ignore
+            UnionByNameVisitor(update_schema=self, existing_schema=self._schema, case_sensitive=self._case_sensitive),

Review Comment:
   Can you revert this change?



##########
pyiceberg/catalog/__init__.py:
##########
@@ -717,6 +791,10 @@ def _get_updated_props_and_update_summary(
 
         return properties_update_summary, updated_properties
 
+    @staticmethod
+    def empty_table_metadata() -> TableMetadata:
+        return TableMetadataV1(location="", last_column_id=-1, schema=Schema())

Review Comment:
   I recommend creating a V2 table by default. This is also the case for Java. Partition evolution is very awkward for V1 tables (keeping the old partitions as null-transforms).



##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(

Review Comment:
   I was leaning towards staged since we use the same name for the REST catalog. I'm not super strong on this one. Let's keep it transaction unless anyone else jumps in here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1524675155


##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> CreateTableTransaction:
+        """Create a CreateTableTransaction.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            CreateTableTransaction: createTableTransaction instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        return CreateTableTransaction(
+            self._create_staged_table(identifier, schema, location, partition_spec, sort_order, properties)

Review Comment:
   Style nit: I would just inline this function, avoids scrolling :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1525833930


##########
pyiceberg/table/metadata.py:
##########
@@ -287,6 +287,31 @@ def _generate_snapshot_id() -> int:
     return snapshot_id
 
 
+class IncompleteTableMetadata(TableMetadataCommonFields):

Review Comment:
   Honestly, I also do not like this solution 😢 . I pushed this just because it looks better than the `InitialTableMetadataConstructor` thing in my first draft. 
   
   In general, we need a way to construct a base metadata without any schema, spec, and sort-order so that we can construct a complete metadata via traversing these [create changes](https://github.com/apache/iceberg-python/pull/498/files#diff-23e8153e0fd497a9212215bd2067068f3b56fa071770c7ef326db3d3d03cee9bR388-R413). Currently, we have some before validators that automatically adds schema (construct_schemas) and spec (construct_partition_specs) to the metadata model. This behavior conflicts with the required create changes: AddSchemaUpdate and AddPartitionSpecUpdate. (https://github.com/apache/iceberg-python/pull/498#discussion_r1513709599)
   
   I am thinking a third approach that we may add a flag to `TableUpdate` to mark it as `create changes` and let `_apply_table_update` handle these updates specially. We will exclude this flag from serialization so RestCatalog is not affected. If it can work, we do not need this additional class anymore.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513259769


##########
pyiceberg/table/__init__.py:
##########
@@ -753,6 +807,143 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda
     return new_metadata.model_copy(deep=True)
 
 
+class InitialTableMetadataConstructor:

Review Comment:
   According to the Rest Catalog Open API spec, only the **format-version** and the **table-uuid** are the only required attributes of the entire Spec. What are your thoughts on updating the TableMetadataCommonFields Fields to align with the Rest Catalog Open API Spec? This will also allow us to create a Base TableMetadata, and use the existing **_apply_table_update** functions instead of having to write a separate constructor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion]CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1513787586


##########
pyiceberg/catalog/glue.py:
##########
@@ -437,46 +471,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:

Review Comment:
   I think the `AssertCreate` is a just safeguard to notify us if the create table commit incorrectly goes into the update table path. i.e. A create table transaction is constructed with an identifier that is already registered in the catalog. In that case, `AssertCreate.validate` raise an exception since the metadata exists
   



##########
pyiceberg/catalog/glue.py:
##########
@@ -437,46 +471,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:

Review Comment:
   I think the `AssertCreate` is a safeguard to notify us if the create table commit incorrectly goes into the update table path. i.e. A create table transaction is constructed with an identifier that is already registered in the catalog. In that case, `AssertCreate.validate` raise an exception since the metadata exists
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX merged PR #498:
URL: https://github.com/apache/iceberg-python/pull/498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1524665622


##########
pyiceberg/table/__init__.py:
##########
@@ -620,15 +711,17 @@ def _(update: AddPartitionSpecUpdate, base_metadata: TableMetadata, context: _Ta
         update={
             "partition_specs": base_metadata.partition_specs + [update.spec],
             "last_partition_id": max(
-                max(field.field_id for field in update.spec.fields),
+                max([field.field_id for field in update.spec.fields] + [0]),

Review Comment:
   Avoids a list concatenation :)
   ```suggestion
                   max(0, *[field.field_id for field in update.spec.fields]),
   ```



##########
pyiceberg/catalog/glue.py:
##########
@@ -435,46 +439,69 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
         )
         database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
 
-        current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
-        glue_table_version_id = current_glue_table.get("VersionId")
-        if not glue_table_version_id:
-            raise CommitFailedException(f"Cannot commit {database_name}.{table_name} because Glue table version id is missing")
-        current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
-        base_metadata = current_table.metadata
-
-        # Validate the update requirements
-        for requirement in table_request.requirements:
-            requirement.validate(base_metadata)
-
-        updated_metadata = update_table_metadata(base_metadata, table_request.updates)
-        if updated_metadata == base_metadata:
-            # no changes, do nothing
-            return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
-
-        # write new metadata
-        new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
-        new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
-        self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
-
-        update_table_input = _construct_table_input(
-            table_name=table_name,
-            metadata_location=new_metadata_location,
-            properties=current_table.properties,
-            metadata=updated_metadata,
-            glue_table=current_glue_table,
-            prev_metadata_location=current_table.metadata_location,
-        )
+        try:
+            current_glue_table = self._get_glue_table(database_name=database_name, table_name=table_name)
+            # Update the table
+            glue_table_version_id = current_glue_table.get("VersionId")
+            if not glue_table_version_id:
+                raise CommitFailedException(
+                    f"Cannot commit {database_name}.{table_name} because Glue table version id is missing"
+                )
+            current_table = self._convert_glue_to_iceberg(glue_table=current_glue_table)
+            base_metadata = current_table.metadata
+
+            # Validate the update requirements
+            for requirement in table_request.requirements:
+                requirement.validate(base_metadata)
+
+            updated_metadata = update_table_metadata(base_metadata, table_request.updates)
+            if updated_metadata == base_metadata:
+                # no changes, do nothing
+                return CommitTableResponse(metadata=base_metadata, metadata_location=current_table.metadata_location)
+
+            # write new metadata
+            new_metadata_version = self._parse_metadata_version(current_table.metadata_location) + 1
+            new_metadata_location = self._get_metadata_location(current_table.metadata.location, new_metadata_version)
+            self._write_metadata(updated_metadata, current_table.io, new_metadata_location)
+
+            update_table_input = _construct_table_input(
+                table_name=table_name,
+                metadata_location=new_metadata_location,
+                properties=current_table.properties,
+                metadata=updated_metadata,
+                glue_table=current_glue_table,
+                prev_metadata_location=current_table.metadata_location,
+            )
 
-        # Pass `version_id` to implement optimistic locking: it ensures updates are rejected if concurrent
-        # modifications occur. See more details at https://iceberg.apache.org/docs/latest/aws/#optimistic-locking
-        self._update_glue_table(
-            database_name=database_name,
-            table_name=table_name,
-            table_input=update_table_input,
-            version_id=glue_table_version_id,
-        )
+            # Pass `version_id` to implement optimistic locking: it ensures updates are rejected if concurrent
+            # modifications occur. See more details at https://iceberg.apache.org/docs/latest/aws/#optimistic-locking
+            self._update_glue_table(
+                database_name=database_name,
+                table_name=table_name,
+                table_input=update_table_input,
+                version_id=glue_table_version_id,
+            )
+
+            return CommitTableResponse(metadata=updated_metadata, metadata_location=new_metadata_location)
+        except NoSuchTableError:
+            # Create the table
+            updated_metadata = construct_table_metadata(table_request.updates)

Review Comment:
   I would expect `create_table` to be called from the `_commit` from the `Transaction`. 



##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(

Review Comment:
   WDYT?
   ```suggestion
       def create_table_staged(
   ```



##########
pyiceberg/table/metadata.py:
##########
@@ -287,6 +287,31 @@ def _generate_snapshot_id() -> int:
     return snapshot_id
 
 
+class IncompleteTableMetadata(TableMetadataCommonFields):

Review Comment:
   Do you know if we need this class? I don't like that we must update the return signature when updating the table metadata to `TableMetadata | IncompleteTableMetadata`. While the name looks very similar, the classes have no OOP relationship. When creating a table we need an UUID (which we can generate), location, and an initial schema. Looking at the tests, we have all of those.



##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> CreateTableTransaction:
+        """Create a CreateTableTransaction.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            CreateTableTransaction: createTableTransaction instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        return CreateTableTransaction(
+            self._create_staged_table(identifier, schema, location, partition_spec, sort_order, properties)

Review Comment:
   Style nit: I would just inline this function, avoids scrolling :)



##########
tests/integration/test_writes.py:
##########
@@ -632,6 +632,60 @@ def test_write_and_evolve(session_catalog: Catalog, format_version: int) -> None
                 snapshot_update.append_data_file(data_file)
 
 
+@pytest.mark.integration
+@pytest.mark.parametrize("format_version", [2])
+def test_create_table_transaction(session_catalog: Catalog, format_version: int) -> None:
+    if format_version == 1:
+        pytest.skip(
+            "There is a bug in the REST catalog (maybe server side) that prevents create and commit a staged version 1 table"

Review Comment:
   Interesting, let me track this down.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] [WIP][Discussion] CreateTableTransaction Implementation [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1525837679


##########
tests/integration/test_writes.py:
##########
@@ -632,6 +632,60 @@ def test_write_and_evolve(session_catalog: Catalog, format_version: int) -> None
                 snapshot_update.append_data_file(data_file)
 
 
+@pytest.mark.integration
+@pytest.mark.parametrize("format_version", [2])
+def test_create_table_transaction(session_catalog: Catalog, format_version: int) -> None:
+    if format_version == 1:
+        pytest.skip(
+            "There is a bug in the REST catalog (maybe server side) that prevents create and commit a staged version 1 table"

Review Comment:
   Thanks! My initial guess is that [CatalogHandler](https://github.com/apache/iceberg/blob/f19643a93f5dac99bbdbc9881ef19c89d7bcd3eb/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L336-L342) use the default values to build the initial empty metadata. But the default format-version is now [2](https://github.com/apache/iceberg/blob/6e21bbf4c4cd2c8351a64636f91d05d00492dff2/core/src/main/java/org/apache/iceberg/TableMetadata.java#L911) so it's impossible for the server to build v1 metadata in this case.
   
   I remembered that `iceberg-rest` used the CatalogHandler to wrap the SqlCatalog. I will double-check that this weekend and do more tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1538156050


##########
pyiceberg/catalog/__init__.py:
##########
@@ -288,6 +291,78 @@ def __init__(self, name: str, **properties: str):
     def _load_file_io(self, properties: Properties = EMPTY_DICT, location: Optional[str] = None) -> FileIO:
         return load_file_io({**self.properties, **properties}, location)
 
+    def _create_staged_table(
+        self,
+        identifier: Union[str, Identifier],
+        schema: Union[Schema, "pa.Schema"],
+        location: Optional[str] = None,
+        partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
+        sort_order: SortOrder = UNSORTED_SORT_ORDER,
+        properties: Properties = EMPTY_DICT,
+    ) -> StagedTable:
+        """Create a table and return the table instance without committing the changes.
+
+        Args:
+            identifier (str | Identifier): Table identifier.
+            schema (Schema): Table's schema.
+            location (str | None): Location for the table. Optional Argument.
+            partition_spec (PartitionSpec): PartitionSpec for the table.
+            sort_order (SortOrder): SortOrder for the table.
+            properties (Properties): Table properties that can be a string based dictionary.
+
+        Returns:
+            Table: the created table instance.
+
+        Raises:
+            TableAlreadyExistsError: If a table with the name already exists.
+        """
+        schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
+
+        database_name, table_name = self.identifier_to_database_and_table(identifier)
+
+        location = self._resolve_table_location(location, database_name, table_name)
+        metadata_location = self._get_metadata_location(location=location)
+        metadata = new_table_metadata(
+            location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
+        )
+        io = load_file_io(properties=self.properties, location=metadata_location)
+        return StagedTable(
+            identifier=(self.name, database_name, table_name),
+            metadata=metadata,
+            metadata_location=metadata_location,
+            io=io,
+            catalog=self,
+        )
+
+    def create_table_transaction(

Review Comment:
   +1 for create_table_transaction 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #498:
URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1542411739


##########
pyiceberg/catalog/rest.py:
##########
@@ -524,7 +537,32 @@ def create_table(
         except HTTPError as exc:
             self._handle_non_200_response(exc, {409: TableAlreadyExistsError})
 
-        table_response = TableResponse(**response.json())
+        return TableResponse(**response.json())
+
+    @retry(**_RETRY_ARGS)
+    def _create_staged_table(

Review Comment:
   Thanks! This is a great suggestion! After going through the current code, I find that not only `_create_staged_table` but also most helper functions in `Catalog` are specific to non-RestCatalog. Hence, I add a new class named `MetastoreCatalog` (appreciate any suggestions on the name) and make all non-RestCatalog inherit from it instead. 
   
   Since it is a big refactoring, please let me know if you want this to happen in a follow-up PR.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org