You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/04/18 04:47:42 UTC
[PR] fix: Iceberg warehouse path mismatch between Python and Java/Scala catalogs [texera]
aglinxinyuan opened a new pull request, #4409:
URL: https://github.com/apache/texera/pull/4409
### What changes were proposed in this PR?
Iceberg tables created via the Python API could not be read back on the Java/Scala side because the two runtimes were registering the Postgres JDBC catalog with different warehouse values, which PyIceberg persists into the table metadata.
The Python side (create_postgres_catalog in amber/src/main/python/core/storage/iceberg/iceberg_utils.py) was prefixing the same path with file://, so tables created by Python UDFs were registered under file:///... while Scala-side lookups expected the un-prefixed path.
This caused subsequent reads of Python-written Iceberg tables to fail (wrong/unresolvable warehouse path in the metadata pointer).
Drop the file:// prefix in create_postgres_catalog so Python matches the Scala catalog's warehouse value exactly. PyIceberg accepts a plain local path here and will treat it as a local filesystem warehouse, consistent with the Scala JdbcCatalog configuration.
### Any related issues, documentation, discussions?
Closes #4408
### How was this PR tested?
1. Create an Iceberg table from a Python UDF operator and confirm it can be read back from the Scala/Java engine in the same workflow.
2. Re-run existing Iceberg-backed workflows (Python-write → Python-read and Python-write → Scala-read) and confirm no regressions.
3. Verify on Windows that the warehouse path passed in (with colon stripped) still resolves correctly from Python.
### Was this PR authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Iceberg warehouse path mismatch between Python and Java/Scala catalogs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #4409:
URL: https://github.com/apache/texera/pull/4409#discussion_r3104677069
##########
amber/src/main/python/core/storage/iceberg/iceberg_utils.py:
##########
@@ -148,7 +148,7 @@ def create_postgres_catalog(
catalog_name,
**{
"uri": f"postgresql+pg8000://{username}:{password}@{uri_without_scheme}",
- "warehouse": f"file://{warehouse_path}",
+ "warehouse": warehouse_path,
},
Review Comment:
added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Iceberg warehouse path mismatch between Python and Java/Scala catalogs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #4409:
URL: https://github.com/apache/texera/pull/4409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Iceberg warehouse path mismatch between Python and Java/Scala catalogs [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4409:
URL: https://github.com/apache/texera/pull/4409#discussion_r3104532872
##########
amber/src/main/python/core/storage/iceberg/iceberg_utils.py:
##########
@@ -148,7 +148,7 @@ def create_postgres_catalog(
catalog_name,
**{
"uri": f"postgresql+pg8000://{username}:{password}@{uri_without_scheme}",
- "warehouse": f"file://{warehouse_path}",
+ "warehouse": warehouse_path,
},
Review Comment:
On Windows, the Java/Scala Postgres JdbcCatalog initializes the warehouse using `warehouse.toString.replace(":", "")` (see `common/workflow-core/src/main/scala/org/apache/texera/amber/util/IcebergUtil.scala:141-145`). Python now passes `warehouse_path` through unmodified, which will likely reintroduce a Python-vs-Scala mismatch for drive-letter paths (e.g., `C:/...` vs `C/...`) and break cross-runtime table reads again. Consider applying the same normalization here (or ensuring the value passed into the Python worker is already normalized) so both runtimes persist the exact same warehouse string.
##########
amber/src/main/python/core/storage/iceberg/iceberg_utils.py:
##########
@@ -148,7 +148,7 @@ def create_postgres_catalog(
catalog_name,
**{
"uri": f"postgresql+pg8000://{username}:{password}@{uri_without_scheme}",
- "warehouse": f"file://{warehouse_path}",
+ "warehouse": warehouse_path,
},
Review Comment:
This change is fixing a cross-runtime compatibility issue by altering the persisted `warehouse` value. There are pytest tests for other helpers in this module (e.g., `core/storage/iceberg/test_iceberg_utils_large_binary.py`), but nothing guarding `create_postgres_catalog` behavior. Adding a unit test that mocks/patches `SqlCatalog` and asserts the `warehouse` argument is passed without the `file://` prefix (and with any required normalization) would help prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org