You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "slachiewicz (via GitHub)" <gi...@apache.org> on 2026/09/04 20:30:53 UTC
[PR] fix(core): parse timestamp keygen filter literals by their own spelling [hudi-rs]
slachiewicz opened a new pull request, #746:
URL: https://github.com/apache/hudi-rs/pull/746
## Description
> **Stacked on #725, which is stacked on #724.** The diff shown here includes both PRs'
> commits and will shrink to the four below once they merge. Review the last four commits
> only: `test(core): add a timestamp-column fixture for the timestamp key generator`,
> `fix(core): parse timestamp keygen filter literals by their own spelling`,
> `fix(core): keep reading when a partition filter cannot be rewritten` and
> `fix(datafusion): forward timestamp keygen source-column predicates to listing`.
`TimestampBasedKeyGenerator` parsed an incoming filter literal with
`hoodie.keygen.timebased.timestamp.type`, which describes how the *source column* stores
its value. A literal is spelled by whoever wrote the query, and the two need not agree. A
table whose partition column is a real `TIMESTAMP` can only declare an epoch type honestly,
since that is what Spark hands the key generator, and then no datetime a caller writes
survives `parse::<i64>()`. The failure was not a degraded scan: the error came out of
`PartitionPruner::new` and the read failed outright.
The reachable paths turned out to be narrower than the description above suggests, and the
new fixture pins which is which. A DataFusion timestamp literal never hit the parser at
all, because `ScalarValue`'s `Display` prints the raw epoch integer in the column's unit,
which is exactly the declared encoding; `test_convert_timestamp_literal_is_epoch_integer_in_its_unit`
pins that spelling since the key generator now depends on it. The string filter API is
the path that failed: `ReadOptions::with_filters(("ts", ">=", "2024-03-01T00:00:00Z"))`,
and with it the Python and C++ bindings, returned
`Failed to parse epoch microseconds '2024-03-01T00:00:00Z'` on the fixture.
**Literals are parsed by their own spelling.** The configured encoding is still tried
first, so every input that works today behaves exactly as before, and the fallback takes
RFC 3339 with an offset, a naive datetime with `T` or space, and a bare date, all as UTC.
The two families are disjoint: no epoch integer parses as a datetime and no datetime
parses as an integer, so the fallback cannot reinterpret an accepted input, and
`test_parse_timestamp_literal_families_are_disjoint` holds that. `input.dateformat` keeps
describing the source column and is not consulted for the fallback. One side effect worth
naming: a `DATE_STRING` table with a time-bearing input format now accepts a bare date
literal, which previously errored because the format demanded a time.
**An unparseable literal no longer fails the read.** Pruning is an optimization, so a
filter the generator cannot rewrite is kept untransformed, the way #724 already handles
`!=` and `NOT IN`. It still names the source column, which the partition schema does not
contain, so it fails to bind and lands in #725's `unapplied_filters` while the read goes
ahead and enforces it per row.
`unparseable_literal_is_reported_as_unapplied_instead_of_failing_the_read` checks that
composition.
**DataFusion never forwarded the predicate.** Once the fixture existed, the DataFusion
query returned the right rows but scanned all four partitions. The provider classified
partition predicates by the partition schema, which for a timestamp key generator holds
only `_hoodie_partition_path`, so a predicate on `ts` was never passed to file listing.
`HudiDataSource` now also lists the key generator's source column for listing purposes.
Exactness is unchanged: a predicate on a derived column is never `Exact`, because a day
partition holds every instant of that day, and the test asserts the equality query still
returns one row rather than the whole day.
The fixture is a table version 9 COW written by Hudi 1.2.0 on Spark 3.5.7 through the
DataFrame writer. Two things ruled out the Spark SQL recipe the other fixtures use: Hudi
0.15.0 has no `EPOCHMICROSECONDS` and formats the micros as millis (year 56134), and Hudi
1.2.0's SQL layer re-parses a timestamp partition path as a long in
`SqlKeyGenerator.convertPartitionPathToSqlType` and throws on `2024-03-01`. Both are
recorded in the fixture's recipe.
Not in this PR: `CustomKeyGenerator` (#549), and the typed-value refactor in #609. That
refactor is the proper end state for this class of bug, since a typed `Filter` would carry
the DataFusion literal without a string round trip at all. This change is the narrow
increment ahead of it because the string API is what the bindings expose today and will
keep exposing after #609, whose plan keeps the string tuple form as the cross-language
entry point, so a parser that reads a literal by its own spelling is needed either way.
Verified: `cargo clippy --all-targets --all-features --workspace -- -D warnings` is clean,
the no-spill `hudi-core` leg passes, and the keygen, partition, `table_read_tests` and
`hudi-datafusion` suites pass locally. `make format check test` did not complete locally
(the machine ran out of disk during the Python build), so CI is the check for that.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
*This change was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): parse timestamp keygen filter literals by their own spelling [hudi-rs]
Posted by "slachiewicz (via GitHub)" <gi...@apache.org>.
slachiewicz commented on PR #746:
URL: https://github.com/apache/hudi-rs/pull/746#issuecomment-5567830356
Parking this as a draft: it needs more work before it is ready for review, and its base has drifted from the rest of the stack, so it does not yet carry the escaping fix it depends on. Picking it back up next week.
*This comment was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): parse timestamp keygen filter literals by their own spelling [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #746:
URL: https://github.com/apache/hudi-rs/pull/746#issuecomment-5561335550
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/746?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `92.13483%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.60%. Comparing base ([`922a62c`](https://app.codecov.io/gh/apache/hudi-rs/commit/922a62cd0ae9af6d0737dd70c4c786ac0de542ff?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a01a7d6`](https://app.codecov.io/gh/apache/hudi-rs/commit/a01a7d69cc97296f873bb191d1f3fe5659fc595f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 21 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/746?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/src/table/partition.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/746?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftable%2Fpartition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RhYmxlL3BhcnRpdGlvbi5ycw==) | 85.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/746?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/datafusion/src/lib.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/746?src=pr&el=tree&filepath=crates%2Fdatafusion%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2RhdGFmdXNpb24vc3JjL2xpYi5ycw==) | 94.11% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/746?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #746 +/- ##
==========================================
+ Coverage 79.53% 79.60% +0.07%
==========================================
Files 126 126
Lines 12301 12419 +118
==========================================
+ Hits 9783 9886 +103
- Misses 2518 2533 +15
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/746?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org