You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anxkhn (via GitHub)" <gi...@apache.org> on 2026/07/05 14:40:58 UTC

[PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

anxkhn opened a new pull request, #3602:
URL: https://github.com/apache/iceberg-python/pull/3602

   
   # Rationale for this change
   
   `DynamoDbCatalog` stores each namespace property under a key prefixed with
   `PROPERTY_KEY_PREFIX` (`"p."`) via `_add_property_prefix`, and decodes it on read
   with `_remove_property_prefix`:
   
   ```python
   def _remove_property_prefix(prop: str) -> str:
       return prop.lstrip(PROPERTY_KEY_PREFIX)
   ```
   
   `str.lstrip` treats its argument as a *set of characters* to strip, not a literal
   prefix. With `PROPERTY_KEY_PREFIX == "p."` it removes every leading `p` and `.`
   from the decoded key, so any namespace property whose name starts with one of
   those characters is corrupted when read back through `load_namespace_properties`:
   
   | stored property | returned key (before) |
   | --- | --- |
   | `provider` | `rovider` |
   | `path` | `ath` |
   | `partition` | `artition` |
   | `ppp` | `` (empty) |
   | `.hidden` | `hidden` |
   
   `update_namespace_properties` reads the same decoded properties, so it operates on
   the corrupted keys too. Property names that do not start with `p` or `.` (such as
   `comment`, `owner`, `location`) round-trip correctly, which is why the existing
   tests, whose fixtures avoid those leading characters, never surfaced the bug.
   
   The fix uses `str.removeprefix`, which strips only the literal `"p."` prefix. The
   single caller (`_get_namespace_properties`) already guards each key with
   `startswith(PROPERTY_KEY_PREFIX)`, so behavior for every valid key is unchanged
   and only the previously corrupted keys are affected. `str.removeprefix` is
   available in Python 3.9+ and this project requires `>=3.10`.
   
   ## Are these changes tested?
   
   Yes. Added `test_load_namespace_properties_with_prefix_char_keys` in
   `tests/catalog/test_dynamodb.py`, a `create_namespace` / `load_namespace_properties`
   round trip (backed by `moto`'s `mock_aws`) over keys `provider`, `path`, `ppp`,
   `.hidden`, and `comment`. The test fails on the previous `lstrip` implementation
   (keys come back corrupted) and passes with `removeprefix`.
   
   `tests/catalog/test_dynamodb.py` passes (40 tests), and `make lint` (ruff,
   ruff-format, mypy, and the other pre-commit hooks) is clean. The DynamoDB catalog
   has no separate integration test path; behavior is covered by the mocked unit test.
   
   ## Are there any user-facing changes?
   
   Yes: a bug fix. Reading namespace properties from a DynamoDB-backed catalog now
   returns property keys that begin with `p` or `.` correctly instead of truncating
   their leading characters. No public API changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

Posted by "anxkhn (via GitHub)" <gi...@apache.org>.
anxkhn commented on PR #3602:
URL: https://github.com/apache/iceberg-python/pull/3602#issuecomment-5092159103

   updated as suggested: removed the one-use helper and builtin-focused tests, then called `removeprefix` directly with a comment explaining the difference from `lstrip`. rebased onto current main, and the dynamodb tests and all hooks pass locally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #3602:
URL: https://github.com/apache/iceberg-python/pull/3602#discussion_r3525663357


##########
tests/catalog/test_dynamodb.py:
##########
@@ -502,6 +502,26 @@ def test_load_namespace_properties(_bucket_initialize: None, database_name: str)
         assert v == test_properties[k]
 
 
+@mock_aws
+def test_load_namespace_properties_with_prefix_char_keys(_bucket_initialize: None, database_name: str) -> None:
+    # Property keys starting with a character in the "p." prefix (e.g. "p" or ".") must
+    # survive a round trip. The stored keys are prefixed with PROPERTY_KEY_PREFIX ("p."),
+    # so decoding must strip only that literal prefix, not any leading "p"/"." characters.
+    test_properties = {
+        "provider": "s3",
+        "path": f"s3://{BUCKET_NAME}/{database_name}.db",
+        "ppp": "3",
+        ".hidden": "yes",
+        "comment": "not affected",
+    }
+    test_catalog = DynamoDbCatalog("test_ddb_catalog")
+    test_catalog.create_namespace(database_name, test_properties)
+    listed_properties = test_catalog.load_namespace_properties(database_name)
+    for k, v in test_properties.items():
+        assert k in listed_properties
+        assert listed_properties[k] == v
+

Review Comment:
   This only asserts expected keys appear in result (subset check), not that result contains no extra keys. We could simply verify the entire properties:
   
   ```py
   assert listed_properties == test_properties
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on PR #3602:
URL: https://github.com/apache/iceberg-python/pull/3602#issuecomment-4888089550

   Do we even need tests? I don't think it is our job to test that the `removeprefix` builtin works.
   
   Additionally, this abstraction over a builtin is only used in one place, would it be simpler to just use `removeprefix` in the only place that [`_remove_property_prefix`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/dynamodb.py#L840C13-L840C36) is used and just have a comment explaining why we are calling`removeprefix`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

Posted by "anxkhn (via GitHub)" <gi...@apache.org>.
anxkhn commented on PR #3602:
URL: https://github.com/apache/iceberg-python/pull/3602#issuecomment-5092119133

   Pushed a revision addressing the latest review/CI feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix: use removeprefix to decode DynamoDB namespace property keys [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #3602:
URL: https://github.com/apache/iceberg-python/pull/3602


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org