You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "geruh (via GitHub)" <gi...@apache.org> on 2025/02/11 07:02:14 UTC

[PR] Add support for `write.metadata.path` [iceberg-python]

geruh opened a new pull request, #1642:
URL: https://github.com/apache/iceberg-python/pull/1642

   Adding support for writing metadata to a custom path set via `write.metadata.path` property. Since the Python library consolidates the table operation classes in both the table and catalog classes, I had to surface the metadata file location handling to the base `Catalog` class to avoid circular dependencies. This way we are also able to centralize the metadata location handling for table metadata and snapshots. 
   
   
   Relates to #1492 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955383770


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   this is interesting because for data file path, we can use `TableProperties.WRITE_DATA_PATH` to override LocationProvider's `self.data_path`, which is the prefix of the data file path.
   And `new_data_location` to override the behavior of the fully-qualified data file location.
   
   We're trying recreate the same behavior for metadata path.
   `TableProperties.WRITE_METADATA_PATH` to override the LocationProvider's `self.metadata_path`, which is the prefix of the metadata file path. 
   And `new_metadata_location` to override the behavior of the fully-qualified metadata file location.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955405522


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+While data files can leverage provider-specific optimizations, metadata files always follow a simple path configuration. Regardless of
+the `LocationProvider` used, metadata files are written to the path specified by the [`write.metadata.path` table configuration](#write-options) table property.

Review Comment:
   based on talking to myself [here](https://github.com/apache/iceberg-python/pull/1642/files#r1955395642)
   
   how about something like,
   ```suggestion
   Both data file and metadata file locations can be customized by configuring the table properties [write.data.path and write.metadata.path](#write-options), respectively.
   
   For more granular control, you can override the `LocationProvider`'s `new_data_location` and `new_metadata_location` methods to define custom logic for generating file paths.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1951181666


##########
pyiceberg/catalog/__init__.py:
##########
@@ -746,6 +747,24 @@ def _convert_schema_if_needed(schema: Union[Schema, "pa.Schema"]) -> Schema:
             pass
         raise ValueError(f"{type(schema)=}, but it must be pyiceberg.schema.Schema or pyarrow.Schema")
 
+    @staticmethod
+    def metadata_file_location(table_location: str, file_name: str, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   nit: should this be a Table static method, instead of a Catalog one? I think Table makes sense to me esp since we're reading TableProperties



##########
pyiceberg/table/update/snapshot.py:
##########
@@ -243,13 +233,8 @@ def _commit(self) -> UpdatesAndRequirements:
         next_sequence_number = self._transaction.table_metadata.next_sequence_number()
 
         summary = self._summary(self.snapshot_properties)
-
-        manifest_list_file_path = _generate_manifest_list_path(
-            location=self._transaction.table_metadata.location,
-            snapshot_id=self._snapshot_id,
-            attempt=0,
-            commit_uuid=self.commit_uuid,
-        )
+        file_name = f"{self.commit_uuid}-m{self._snapshot_id}-a0.avro"
+        manifest_list_file_path = self._transaction._table.metadata_file_location(file_name)

Review Comment:
   is this the same as `_generate_manifest_list_path`? 
   
   `_generate_manifest_list_path` was using
   ```
       return f"{location}/metadata/snap-{snapshot_id}-{attempt}-{commit_uuid}.avro"
   ```
   
   aside from `{location}/metadata` which we are changing, the rest should stay the same. 



##########
pyiceberg/table/update/snapshot.py:
##########
@@ -295,13 +280,9 @@ def new_manifest_writer(self, spec: PartitionSpec) -> ManifestWriter:
         )
 
     def new_manifest_output(self) -> OutputFile:
-        return self._io.new_output(
-            _new_manifest_path(
-                location=self._transaction.table_metadata.location,
-                num=next(self._manifest_num_counter),
-                commit_uuid=self.commit_uuid,
-            )
-        )
+        file_name = f"{self.commit_uuid}-m{next(self._manifest_num_counter)}.avro"

Review Comment:
   this is the same as `_new_manifest_path` 👍 
   ```
       return f"{location}/metadata/{commit_uuid}-m{num}.avro"
   ```



##########
pyiceberg/table/update/snapshot.py:
##########
@@ -243,13 +233,8 @@ def _commit(self) -> UpdatesAndRequirements:
         next_sequence_number = self._transaction.table_metadata.next_sequence_number()
 
         summary = self._summary(self.snapshot_properties)
-
-        manifest_list_file_path = _generate_manifest_list_path(
-            location=self._transaction.table_metadata.location,
-            snapshot_id=self._snapshot_id,
-            attempt=0,
-            commit_uuid=self.commit_uuid,
-        )
+        file_name = f"{self.commit_uuid}-m{self._snapshot_id}-a0.avro"
+        manifest_list_file_path = self._transaction._table.metadata_file_location(file_name)

Review Comment:
   nit: should we keep `_generate_manifest_list_path` and make it generate the file path after `/metadata`, and then just use `.metadata_file_location()` to include the metadata prefix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   This generally looks fine to me (and configuring metadata locations on `LocationProvider`s is cool)!
   
   1. Spitballing (/nits): I wonder if custom location providers that technically didn't subclass `LocationProvider` but would've worked before still would now fail for metadata writing because they don't have these new methods. If this is the case, it's maybe fine because we can expect them to subclass `LocationProvider` as per the docs.
   2. The way this is written makes me wonder whether users might want to customise their metadata locations *under* a table instead of providing a hard-coded path - in a similar way to custom `LocationProvider`s do for data files (maybe with mass updates they can have a large number of json/avro files so they could want to do the object storage hashing inside the metadata folder too to reduce throttling?) . I don't see that strong of a use case FWIW, but if I'm wrong could maybe add this to docs about custom location providers or start a larger discussion. Any thoughts folks?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954915956


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   for 1, we haven't released a version with LocationProvider. so any references going forward should also have the new methods. We probably want to add a note in the [location provider docs](https://github.com/apache/iceberg-python/blob/1c9f177be4bbb5a1bef78b46f69a1a3d56de427d/mkdocs/docs/configuration.md#location-providers) about the new metadata location feature
   
   > The way this is written makes me wonder whether users might want to customise their metadata locations under a table instead of providing a hard-coded path - in a similar way to custom LocationProviders do for data files
   
   does "under a table" mean path relative to the table's base location? The SimpleLocation Provider uses absolute path. The ObjectStoreLocationProvider is special in that it injects some hash after the base location. But i think a custom metadata location provider can do the same. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955395642


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   ok, brushing up on my python... 
   i think the code is fine as is. both `new_metadata_location` and `new_table_metadata_file_location` can provide default implementation and also be overriden by custom implementations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953202171


##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   My first idea would be to make this part of the `LocationProvider`. Any thoughts on that? @geruh @kevinjqliu 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   This generally looks fine to me (and configuring metadata locations on `LocationProvider`s is cool)!
   
   1. Spitballing: I wonder if custom location providers that technically didn't subclass `LocationProvider` but would've worked before still would now fail for metadata writing because they don't have these new methods. If this is the case, it's maybe fine because we can expect them to subclass `LocationProvider` as per the docs.
   2. The way this is written makes me wonder whether users might want to customise their metadata locations *under* a table instead of providing a hard-coded path - in a similar way to custom `LocationProvider`s do for data files. I don't see that strong of a use case FWIW. But if I'm wrong could maybe add this to docs about custom location providers or start a larger discussion. Any thoughts folks?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956168492


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   > I found a related issue in the main repo here: https://github.com/apache/iceberg/issues/6809. Which looks like it didn't get any traction but still holds some value.
   
   Thanks for digging into this - I wasn't aware of that discussion. Quoting from it, "Object storage mode to not just data files but also metadata files" is what I was getting at here. It's interesting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955183564


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+While data files can leverage provider-specific optimizations, metadata files always follow a simple path configuration. Regardless of
+the `LocationProvider` used, metadata files are written to the path specified by the [`write.metadata.path` table configuration](#write-options) table property.

Review Comment:
   (Being pedantic) As it stands, I think a custom `LocationProvider` can reasonably override (*re-declare* really, because it's Python) the metadata-related methods. Maybe relates to [this discussion](https://github.com/apache/iceberg-python/pull/1642/files#r1954666388)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955110870


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   @smaheshwar-pltr Regarding point 1, I had the same question while working on this. Centralizing locations/paths would definitely make future changes easier, especially with the ongoing discussions around relative path work. I found a related issue in the main repo here: https://github.com/apache/iceberg/issues/6809. Which looks like it didn't get any traction but still holds some value.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956008322


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,16 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+Both data file and metadata file locations can be customized by configuring the table properties [write.data.path and write.metadata.path](#write-options), respectively.

Review Comment:
   ```suggestion
   Both data file and metadata file locations can be customized by configuring the table properties [`write.data.path` and `write.metadata.path`](#write-options), respectively.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   This generally looks fine to me (and configuring metadata locations on `LocationProvider`s is cool)!
   
   1. Spitballing: I wonder if custom location providers that technically didn't subclass `LocationProvider` but would've worked before still would now fail for metadata writing because they don't have these new methods. If this is the case, it's maybe fine because we can expect them to subclass `LocationProvider` as per the docs.
   2. The way this is written makes me wonder whether users might want to customise their metadata locations *under* a table instead of providing a hard-coded path - in a similar way to custom `LocationProvider`s do for data files (maybe with mass updates they can have a large number of json/avro files so they could want to do the object storage hashing inside the metadata folder too to reduce throttling?) . I don't see that strong of a use case FWIW, but if I'm wrong could maybe add this to docs about custom location providers or start a larger discussion. Any thoughts folks?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955380186


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   oh i see the semantic difference. 
   
   `new_data_location` is an abstract method, which allows custom LocationProviders to override. 
   `load_location_provider` returns `SimpleLocationProvider` by default, which then use the `SimpleLocationProvider`'s `new_data_location` method in most cases. 
   
   we can do the same for `new_metadata_location`, making it an abstract method so that custom LocationProviders can override. but also provide a default implementation for both `SimpleLocationProvider` and `ObjectStoreLocationProvider` 
   
   we can then mention in the LocationProvider documentation that `new_metadata_location` can be overriden 
   
   Does this make sense?  @smaheshwar-pltr is this what you were pointing out? 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953384729


##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   Hey @Fokko, yeah it makes sense from an architectural perspective to have the LocationProvider handle all file locations under a table. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956148860


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,16 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.

Review Comment:
   Following the other discussion I think "Apache Iceberg ... managing metadata files" is a bit misleading given it's just PyIceberg. How about:
   
   ```suggestion
   Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data files. In PyIceberg, the
   `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases, and to additionally determine metadata file locations. The
   `LocationProvider` for a table can be specified through table properties.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1951696715


##########
pyiceberg/table/update/snapshot.py:
##########
@@ -84,14 +84,14 @@
     from pyiceberg.table import Transaction
 
 
-def _new_manifest_path(location: str, num: int, commit_uuid: uuid.UUID) -> str:
-    return f"{location}/metadata/{commit_uuid}-m{num}.avro"
+def _new_manifest_file_name(num: int, commit_uuid: uuid.UUID) -> str:
+    return f"{commit_uuid}-m{num}.avro"
 
 
-def _generate_manifest_list_path(location: str, snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:
+def _generate_manifest_list_file_name(snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:

Review Comment:
   nit:
   ```suggestion
   def _new_manifest_list_file_name(snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:
   ```



##########
pyiceberg/table/__init__.py:
##########
@@ -1212,6 +1213,23 @@ def to_daft(self) -> daft.DataFrame:
 
         return daft.read_iceberg(self)
 
+    @staticmethod
+    def metadata_file_location(table_location: str, file_name: str, properties: Properties = EMPTY_DICT) -> str:
+        """Get the full path for a metadata file.
+
+        Args:
+            table_location (str): The base table location
+            file_name (str): Name of the metadata file
+            properties (Properties): Table properties that may contain custom metadata path
+
+        Returns:
+            str: Full path where the metadata file should be stored
+        """

Review Comment:
   nit: what if we model it like `new_data_location`
   https://github.com/apache/iceberg-python/blob/509713bbc23468008eb9e5ac659db9dbb6828c6d/pyiceberg/table/locations.py#L55-L65
   
   ```suggestion
       @staticmethod
       def new_metadata_location(table_location: str, file_name: str, properties: Properties = EMPTY_DICT) -> str:
           """Return a fully-qualified metadata file location for the given filename.
   
           Args:
               table_location (str): The base table location
               file_name (str): Name of the metadata file
               properties (Properties): Table properties that may contain custom metadata path
   
           Returns:
               str:  A fully-qualified location URI for the metadata file.
           """
   ```



##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   do you think we can just get rid of this function entirely and replace with the underlying table function? (`Table.metadata_file_location`)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953257793


##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   i just assume that LocationProvider was strictly for data location, but i dont see why not :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955044524


##########
pyiceberg/table/locations.py:
##########
@@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti
             str: A fully-qualified location URI for the data file.
         """
 
+    def new_table_metadata_file_location(self, new_version: int = 0) -> str:

Review Comment:
   Yeah exactly, I was just curious about user demand for a custom metadata location provider. It exceeds the capability of Java custom location providers and posing the question of whether it makes sense for other implementations.
   
   To clarify, no objections from my end. Updating the docs sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955183564


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+While data files can leverage provider-specific optimizations, metadata files always follow a simple path configuration. Regardless of
+the `LocationProvider` used, metadata files are written to the path specified by the [`write.metadata.path` table configuration](#write-options) table property.

Review Comment:
   As it stands, I think a custom `LocationProvider` can reasonably override (*re-declare* really, because it's Python) the metadata-related methods. Maybe relates to [this discussion](https://github.com/apache/iceberg-python/pull/1642/files#r1954666388)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953356977


##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   Yes, I'm leaning towards adding this to the `LocationProvider`. WDYT @geruh?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#issuecomment-2652211414

   cc @Fokko @smaheshwar-pltr


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956153130


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,16 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+Both data file and metadata file locations can be customized by configuring the table properties [write.data.path and write.metadata.path](#write-options), respectively.
+
+For more granular control, you can override the `LocationProvider`'s `new_data_location` and `new_metadata_location` methods to define custom logic for generating file paths.

Review Comment:
   Nit: This only applies to custom location providers. *We* understand that and that they can be loaded but I think a new reader doesn't have that context at this point. How about appending something like " - see loading custom location providers" and linking `#loading-a-custom-location-provider` there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956481300


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,16 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+Both data file and metadata file locations can be customized by configuring the table properties [write.data.path and write.metadata.path](#write-options), respectively.
+
+For more granular control, you can override the `LocationProvider`'s `new_data_location` and `new_metadata_location` methods to define custom logic for generating file paths.

Review Comment:
   ```suggestion
   For more granular control, you can override the `LocationProvider`'s `new_data_location` and `new_metadata_location` methods to define custom logic for generating file paths. See [`Loading a Custom Location Provider`](configuration.md#loading-a-custom-location-provider).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#issuecomment-2659906743

   Thanks @geruh for the contribution! And thanks @smaheshwar-pltr @Fokko for the review! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1956148860


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,16 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.

Review Comment:
   Following the other discussion I think "Apache Iceberg ... manage metadata files" is a bit misleading given it's just PyIceberg that does this (for now). How about:
   
   ```suggestion
   Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data files. In PyIceberg, the
   `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases, and to additionally determine metadata file locations. The
   `LocationProvider` for a table can be specified through table properties.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955405522


##########
mkdocs/docs/configuration.md:
##########
@@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya
 
 ## Location Providers
 
-Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data. In PyIceberg, the
+Apache Iceberg uses the concept of a `LocationProvider` to manage file paths for a table's data and metadata files. In PyIceberg, the
 `LocationProvider` module is designed to be pluggable, allowing customization for specific use cases. The
 `LocationProvider` for a table can be specified through table properties.
 
-PyIceberg defaults to the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider), which generates
-file paths that are optimized for object storage.
+While data files can leverage provider-specific optimizations, metadata files always follow a simple path configuration. Regardless of
+the `LocationProvider` used, metadata files are written to the path specified by the [`write.metadata.path` table configuration](#write-options) table property.

Review Comment:
   ```suggestion
   Both data file and metadata file locations can be customized by configuring the table properties [write.data.path and write.metadata.path](#write-options), respectively.
   
   For more granular control, you can override the `LocationProvider`'s `new_data_location` and `new_metadata_location` methods to define custom logic for generating file paths.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#issuecomment-2659867627

   Thanks for the great suggestions, i added them and double check the docs rendering
   
   ![Screenshot 2025-02-14 at 9 14 56 AM](https://github.com/user-attachments/assets/ecdeba9b-63dd-4f2e-8905-df13d5cd7954)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1951696715


##########
pyiceberg/table/update/snapshot.py:
##########
@@ -84,14 +84,14 @@
     from pyiceberg.table import Transaction
 
 
-def _new_manifest_path(location: str, num: int, commit_uuid: uuid.UUID) -> str:
-    return f"{location}/metadata/{commit_uuid}-m{num}.avro"
+def _new_manifest_file_name(num: int, commit_uuid: uuid.UUID) -> str:
+    return f"{commit_uuid}-m{num}.avro"
 
 
-def _generate_manifest_list_path(location: str, snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:
+def _generate_manifest_list_file_name(snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:

Review Comment:
   nit: similar naming scheme as the functino above 
   
   ```suggestion
   def _new_manifest_list_file_name(snapshot_id: int, attempt: int, commit_uuid: uuid.UUID) -> str:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1951707200


##########
pyiceberg/table/__init__.py:
##########
@@ -1212,6 +1213,23 @@ def to_daft(self) -> daft.DataFrame:
 
         return daft.read_iceberg(self)
 
+    @staticmethod
+    def metadata_file_location(table_location: str, file_name: str, properties: Properties = EMPTY_DICT) -> str:
+        """Get the full path for a metadata file.
+
+        Args:
+            table_location (str): The base table location
+            file_name (str): Name of the metadata file
+            properties (Properties): Table properties that may contain custom metadata path
+
+        Returns:
+            str: Full path where the metadata file should be stored
+        """
+        if metadata_path := properties.get(TableProperties.WRITE_METADATA_PATH):
+            return f"{metadata_path.rstrip("/")}/{file_name}"

Review Comment:
   ```suggestion
               return f"{metadata_path.rstrip('/')}/{file_name}"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Add support for `write.metadata.path` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1642:
URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1953257793


##########
pyiceberg/catalog/__init__.py:
##########
@@ -929,11 +931,12 @@ def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) ->
         ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
 
     @staticmethod
-    def _get_metadata_location(location: str, new_version: int = 0) -> str:
+    def _get_metadata_location(table_location: str, new_version: int = 0, properties: Properties = EMPTY_DICT) -> str:

Review Comment:
   i just assume that [LocationProvider was strictly for data location](https://github.com/apache/iceberg/blob/8839c9bf1f1d8c9b718f9766302ff8a2018e515f/core/src/main/java/org/apache/iceberg/LocationProviders.java#L37), but i dont see why not :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org