You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhengguangzhuo (via GitHub)" <gi...@apache.org> on 2026/09/29 10:20:30 UTC
[PR] [python] Align DATE partition paths with Java [paimon]
zhengguangzhuo opened a new pull request, #10319:
URL: https://github.com/apache/paimon/pull/10319
Fixes #10034
## Summary
- Align PyPaimon DATE partition data-file paths with Java when partition.legacy-name=true.
- Convert only DATE partition values to epoch days, so 1970-01-02 is written under day=1.
- Apply the same compatibility path to local data files, external data files, bucket indexes, and commit recovery.
## Compatibility
- Existing Python-written ISO directories such as day=1970-01-02 remain readable.
- partition.legacy-name=false keeps the existing ISO naming behavior.
- Non-DATE partition components keep their existing PyPaimon formatting.
- This is a compatibility-focused rework of the DATE path issue after #10045 was closed because its broader implementation introduced breaking changes.
## Verification
- pypaimon/tests/date_partition_path_test.py: 5 passed.
- Focused writer/commit regression tests: 3 passed.
- pycodestyle, pyflakes, compileall, and git diff --check passed.
- The Java Maven test was attempted, but dependency resolution was blocked by a timeout from the configured Maven mirror in the local environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-6073937918
Thanks for the update. I independently tested `1156834035` against `80c5ab9593`. The PR's Python reader passed all 256 tables in my cross-version, mixed-writer matrix, including the original composite-partition failure. Java still cannot read some historical Python layouts; that is an existing limitation, but the compatibility boundary needs to be explicit.
The OSS performance regression is a concern. With 128 small Parquet files, median planning + full-read time increased from **1.71 to 2.24 seconds (+31%)** for a DATE composite partition, and from **1.47 to 2.47 seconds (+68%)** for an escaped STRING partition without DATE. These were three measured runs after warm-up on the same frozen files. Local full-read performance was roughly unchanged. The head adds path-existence checks even for canonical files, so the description's claim of no additional lookups needs updating.
I do not think this is ready to merge. I closed my own attempt in #10045 because these compatibility and performance tradeoffs were unresolved. Please align on the design before making further code changes: which version combinations we support, how readers locate historical files, and what lookup cost is acceptable. Adding another fallback for each reported failure is not a sufficient compatibility strategy, especially when it slows down tables that already use canonical paths. Once the design is agreed, please implement it and provide reproducible correctness and performance results.
Please also change the added Chinese comments and docstrings to English, consistent with the surrounding code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-5908109284
Thanks for working on this issue. I reported it and also attempted a fix in #10045, which I later closed because of compatibility concerns. I think this implementation still misses some of the difficulties I encountered.
There is a read-after-write regression with composite partitions. With `partition.legacy-name=true`, `day=1970-01-02`, and `region=a/b`, this version writes to `day=1/region=a/b/...`, but the Python reader cannot resolve that mixed-format path and fails with `FileNotFoundError`. The added composite-partition test checks the path without verifying readback.
The historical-directory test also writes with the new code and then renames the entire directory. That doesn’t cover historical and new paths coexisting in the same logical partition/bucket, or compatibility between client versions.
We need a compatibility matrix covering Java and Python writers/readers, old and new versions, and both partition naming modes, with the full Cartesian product of supported combinations. This should include different writers appending to the same partition/bucket, verifying that readers return all expected rows, and checking reads after rollback.
I previously tried adding compatibility handling for different path formats, but the additional lookups caused performance regressions. We haven’t settled on how to handle that tradeoff, so this change needs performance evaluation alongside the compatibility tests.
Could you address these gaps before we proceed with the write-path change? Preserving reads of historical ISO directories alone isn’t enough to establish compatibility.
Please also use English for the added comments and docstrings, consistent with the surrounding code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-6135839850
The current head still has end-to-end compatibility value; I reran 51 focused partition/shard/REST FileIO tests and the real Java writer → Python reader/append → Java reader/rollback chain in both naming modes, with standard JDK 8 Maven checks. These passed. I also repeated the exact-baseline/current resolver comparison with production RESTTokenFileIO → PyArrowFileIO delegation and a counted, present-object backend: 128 canonical files add 0→128 get_file_info calls for DATE+escaped partitions and escaped STRING partitions, while plain STRING stays 0→0. This measures calls, not OSS latency. The previously reported production performance gate remains unresolved.
For the strategy question, I recommend preserving a probe-free canonical fast path and explicitly opting affected tables into historical-layout compatibility, or adding a reliable per-file layout signal. A table-wide inferred writer version is insufficient for mixed layouts: current DataFileMeta has no historical/canonical directory marker, and schema IDs or the latest snapshot writer cannot identify the origin of every retained file. An explicit compatibility mode can probe each mixed file while new/already-migrated tables avoid the cost. Please cover the transition and retained snapshots/tags/rollback as well as mixed append.
The current mixed tests establish current Java ↔ current Python only. Please declare and test the intended historical client/version matrix; synthetic old-directory coverage does not establish historical Java-reader support. I cannot infer a project-wide version floor or an accepted OSS latency budget from the current implementation. Until that boundary is explicit, keep this as a supported compatibility proposal with the existing production gate, rather than claiming broad historical interoperability or accepting the measured lookup cost implicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-5934382634
Confirmed the composite-partition read-after-write issue at `7557c7e144` with an actual table write, commit and read. The DATE compatibility problem in #10034 is worth fixing, but this version should be blocked from production.
**[P1] Make the writer and reader agree on the new composite partition layout**
`pypaimon/utils/file_store_path_factory.py:247-273` converts only DATE components, creating a third layout that neither reader path resolves. With default `partition.legacy-name=true`, `day=1970-01-02` and `region=a/b`:
```text
New writer: day=1/region=a/b/bucket-0/data-....parquet
Historical reader path: day=1970-01-02/region=a/b/bucket-0/data-....parquet
Canonical reader path: day=1/region=a%2Fb/bucket-0/data-....parquet
```
The commit succeeds, then readback raises `FileNotFoundError`. An in-process control retaining the previous writer path reads the same row successfully; `partition.legacy-name=false` also reads back successfully. This confirms a newly introduced failure rather than an existing unreadable table.
I ran the added DATE tests plus file-store-commit and Mosaic writer regressions: 39 tests and 2 subtests passed; changed files passed Flake8. The current composite test asserts only the path and misses the failing readback.
Please make planning resolve the actual new layout, or use a fully canonical layout with historical compatibility, and add write → commit → read assertions for composite DATE partitions with values requiring escaping. Preserve historical/new-file coexistence while doing so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "zhengguangzhuo (via GitHub)" <gi...@apache.org>.
zhengguangzhuo commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-6075314761
Thanks for the detailed cross-version results and OSS measurements. I agree this PR is not ready to merge, and I will hold further code changes until we align on the compatibility scope and performance trade-off.
On the current head (`0e1ac11`), CI run [37882502997](https://github.com/apache/paimon/actions/runs/37882502997) completed successfully. All Python jobs passed; Native CI reported 9,586 passed and 112 skipped. The standalone Java job was skipped. However, the Python 3.10 job did run the mixed Java-Python suite: the Java composite DATE write, Python composite DATE read and append, and Java read-after-write/rollback tests passed for both partition naming modes. This validates those current-version combinations, but not the full historical-client matrix or every historical Python layout.
I appreciate your reported 256-table cross-version mixed-writer matrix; I have not independently reproduced it. The historical-layout test currently in this PR constructs the old directory layout synthetically rather than running each supported historical client version. The support boundary for Java readers of historical Python layouts also remains unresolved; your report indicates that some of those combinations are not currently supported.
I also need to correct the performance statement in the PR description. The current change adds canonical-path `exists_batch` checks when the canonical and legacy bucket paths differ, followed by a historical-path lookup when the canonical candidates are absent. `exists_batch` batches paths at the FileIO API, but that alone does not establish the number or cost of metadata requests made by an object-store backend. I have not independently reproduced your OSS benchmark, and local-filesystem timings are not comparable evidence. I will not use them to dismiss the reported regressions of +31% and +68%.
Before I make further code changes, could we agree on:
1. The exact Java/Python versions and writer-to-reader combinations in scope, for both `partition.legacy-name` modes, including mixed-writer append and rollback scenarios—and which combinations are explicitly out of scope.
2. Which historical Python layouts must be readable, and by which readers, especially whether Java reading those layouts is a requirement.
3. An acceptable object-store metadata-lookup/latency budget and a reproducible benchmark setup. In particular, should reads of an already-canonical layout incur any existence probes?
Once we agree on these boundaries and the performance criterion, I can update the compatibility tests and performance description to match. I will also keep new comments and docstrings in English, consistent with the surrounding code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Align DATE partition paths with Java [paimon]
Posted by "zhengguangzhuo (via GitHub)" <gi...@apache.org>.
zhengguangzhuo commented on PR #10319:
URL: https://github.com/apache/paimon/pull/10319#issuecomment-6121201997
Thanks for the latest validation. I updated the PR description to distinguish the maintainer-reported OSS latency measurements on 1156834035 (compared with 80c5ab9593) from the source-free metadata-call comparison on the current 0e1ac11 head.
Before further code changes, could we choose and document one of these strategies?
1. **Metadata-directed reads:** Keep the canonical-layout fast path free of new metadata probes, using a reliable layout signal only if it can also represent historical and canonical files coexisting in the same partition/bucket. Please confirm whether suitable table/partition metadata already exists, and how mixed layouts and migration would be represented.
2. **Compatibility probing:** Retain canonical/historical existence probes for mixed-layout compatibility, but explicitly accept a quantified object-store cost. On 0e1ac11, the source-free call-count comparison observed 128 additional backend `get_file_info` calls for 128 already-canonical DATE-composite files, and the same count for escaped STRING files without DATE; this isolates calls, not latency. The separate OSS report on 1156834035 measured +31% and +68% median planning-plus-full-read time for those workloads. For acceptance, I propose rerunning baseline and candidate on the same frozen OSS data, with the same warm-up and three measured runs, reporting both planning-plus-full-read latency and backend metadata-call counts against an agreed budget.
Could you confirm the supported Java/Python version floor and writer→reader combinations for both `partition.legacy-name` modes, including mixed-writer append and rollback; which historical Python layouts/readers are in scope (especially Java reading those layouts); and whether strategy 1 or 2 is acceptable? I will hold further implementation changes until we agree on the compatibility boundary and performance criterion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org