You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "slachiewicz (via GitHub)" <gi...@apache.org> on 2026/09/04 18:33:58 UTC
[PR] fix(core): fix partition path parsing and report unapplied filters [hudi-rs]
slachiewicz opened a new pull request, #725:
URL: https://github.com/apache/hudi-rs/pull/725
## Description
> **Stacked on #724.** The diff shown here includes that PR's three commits and will shrink
> to the two below once it merges. Review the last two commits only:
> `fix(core): decode partition values after splitting, and honour the Hive null sentinel`
> and `feat(core): report partition filters that could not be applied`.
Two independent correctness problems in partition path handling, plus the guard that would
have made #580 visible instead of silent.
**Decoding ran before splitting.** `parse_segments` percent-decoded the whole path and then
split it on `/` and `=`. Hudi escapes the partition *value* only and leaves the structural
separators literal, per `PartitionPathEncodeUtils.escapePathName` in
`KeyGenUtils.getRecordPartitionPath`, so a value written as `a%2Fb` was decoded to `a/b` and
then torn into two segments. A url-encoded table with any escaped separator in a value could
not be read at all. Splitting now happens first and each value is decoded exactly once.
The url-encoding test this replaces escaped the separators too, which no Hudi writer does,
so it passed against input that cannot occur. The replacements cover escaped `/`, `=`, space
and `%` inside real segments, and pin that `%252F` decodes to `%2F` rather than to `/`.
**The Hive null sentinel was parsed as a value.** `__HIVE_DEFAULT_PARTITION__` was cast to
the partition column's type, which errors on anything numeric or temporal. A parse error is
treated as "keep the partition", so such a table quietly degraded to a full scan. It is now
read as null and compared under SQL three-valued logic, so a comparison against it is
unknown and the partition is retained deliberately rather than by accident.
**Unapplied filters are now reported.** A filter that failed to bind to the partition schema
was dropped with no record. That is correct for a data-column predicate, which is enforced
per row, but it is also how a *partition* predicate disappears: when a key generator cannot
be built, the untransformed filter still names the source column, which the partition schema
does not contain. The result was a full scan that returned no error and could not be told
apart from a successful one. This is the same failure mode as #580, reached by a different
route, and #581 fixed only the ordering cause.
`PartitionPruner` now records those filters and whether the key generator was unavailable,
exposed as `unapplied_filters` and `is_keygen_config_unavailable`, and logs each drop. A
caller that needs pruning to actually happen, rather than silently degrade, can assert on
them. Worth knowing for the second one: the `hoodie.keygen.timebased.*` options are
writer-side and are not guaranteed to reach `hoodie.properties`, so a table can name
`TimestampBasedKeyGenerator` while carrying nothing that says which granularity, timezone or
format it used. Guessing from the class name is unsafe, so pruning is skipped and reported
instead of approximated.
Verified: `make format check test` passes, and `cargo clippy --all-targets --all-features
--workspace -- -D warnings` is clean.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): fix partition path parsing and report unapplied filters [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #725:
URL: https://github.com/apache/hudi-rs/pull/725#issuecomment-5545465225
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/725?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.78261%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.51%. Comparing base ([`922a62c`](https://app.codecov.io/gh/apache/hudi-rs/commit/922a62cd0ae9af6d0737dd70c4c786ac0de542ff?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`894cbeb`](https://app.codecov.io/gh/apache/hudi-rs/commit/894cbeba2bfb0d6bd4bb7e45bca29e68147f7a54?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/725?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/src/table/partition.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/725?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftable%2Fpartition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RhYmxlL3BhcnRpdGlvbi5ycw==) | 80.55% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #725 +/- ##
==========================================
- Coverage 79.53% 79.51% -0.02%
==========================================
Files 126 126
Lines 12301 12311 +10
==========================================
+ Hits 9783 9789 +6
- Misses 2518 2522 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/725?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org