You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hanbings (via GitHub)" <gi...@apache.org> on 2025/02/24 09:39:18 UTC
[PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
hanbings opened a new pull request, #302:
URL: https://github.com/apache/hudi-rs/pull/302
## Description
Added support for ISO8601 time format as described in issues, and overwrote other `*as_of()` methods in the same module.
closes #261
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#issuecomment-2800102113
hi @hanbings sorry for the delay - the issue should have been resolved, could you rebase master please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "hanbings (via GitHub)" <gi...@apache.org>.
hanbings commented on code in PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#discussion_r2051447821
##########
crates/core/src/table/mod.rs:
##########
@@ -474,6 +481,36 @@ impl Table {
}
}
+/// Formats a timestamp string to `yyyyMMddHHmmSSSSS` format.
+///
+/// The function attempts to parse the input timestamp string (`timestamp`) into a `DateTime` object.
+/// It first tries parsing the ISO 8601 format, and if that fails, it tries two other formats: `yyyyMMddHHmmSSSSS`
+/// and `yyyyMMddHHmmSS`. If none of the formats match, it returns the original input string.
+///
+/// # Arguments
+/// - `timestamp`: The input timestamp string to be formatted.
+///
+/// # Returns
+/// A string formatted as `yyyyMMddHHmmSSSSS`. If the input cannot be parsed, the original string is returned.
+fn format_timestamp(timestamp: &str) -> String {
+ if let Ok(datetime) = DateTime::parse_from_rfc3339(timestamp) {
+ return datetime.format("%Y%m%d%H%M%S%3f").to_string();
+ }
+
+ let formats = ["yyyyMMddHHmmSSSSS", "yyyyMMddHHmmSS"];
+ for format in formats.iter() {
+ if let Ok(datetime) = DateTime::parse_from_str(timestamp, format) {
+ return datetime.format("%Y%m%d%H%M%S%3f").to_string();
+ }
+ }
+
+ if timestamp.len() == 10 && timestamp.chars().all(|c| c.is_digit(10) || c == '-') {
Review Comment:
This check is for short time formats like `2019-01-23`. I haven’t come up with a better way to handle this yet — do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "hanbings (via GitHub)" <gi...@apache.org>.
hanbings commented on code in PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#discussion_r2051447568
##########
crates/core/src/table/mod.rs:
##########
@@ -474,6 +481,36 @@ impl Table {
}
}
+/// Formats a timestamp string to `yyyyMMddHHmmSSSSS` format.
+///
+/// The function attempts to parse the input timestamp string (`timestamp`) into a `DateTime` object.
+/// It first tries parsing the ISO 8601 format, and if that fails, it tries two other formats: `yyyyMMddHHmmSSSSS`
+/// and `yyyyMMddHHmmSS`. If none of the formats match, it returns the original input string.
+///
+/// # Arguments
+/// - `timestamp`: The input timestamp string to be formatted.
+///
+/// # Returns
+/// A string formatted as `yyyyMMddHHmmSSSSS`. If the input cannot be parsed, the original string is returned.
+fn format_timestamp(timestamp: &str) -> String {
Review Comment:
I have added an is_utc parameter to this function in the new commit to mark whether the utc time zone is used, and changed the return value to `Result`, it will return a CoreError::Timeline when parsing errors occur.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "hanbings (via GitHub)" <gi...@apache.org>.
hanbings commented on PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#issuecomment-2726871818
It seems that there are some dependency conflicts here (in Github Actions).
How should we proceed with the next step?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#issuecomment-2800105341
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/302?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `61.11111%` with `7 lines` in your changes missing coverage. Please review.
> Project coverage is 89.73%. Comparing base [(`f73609c`)](https://app.codecov.io/gh/apache/hudi-rs/commit/f73609cd15ed105e2b904ae8f20b6f3c2182cfd7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`f1cee54`)](https://app.codecov.io/gh/apache/hudi-rs/commit/f1cee5400afdee6df052885bb831a341a8106625?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/302?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/src/table/mod.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/302?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ftable%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3RhYmxlL21vZC5ycw==) | 61.11% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/302?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #302 +/- ##
==========================================
- Coverage 89.81% 89.73% -0.08%
==========================================
Files 41 41
Lines 2130 2144 +14
==========================================
+ Hits 1913 1924 +11
- Misses 217 220 +3
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/302?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support more timestamp formats for time-travel and incremental APIs [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #302:
URL: https://github.com/apache/hudi-rs/pull/302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: iso8601 support in Time-travel & Incremental API [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #302:
URL: https://github.com/apache/hudi-rs/pull/302#discussion_r2041204480
##########
crates/core/src/table/mod.rs:
##########
@@ -474,6 +481,36 @@ impl Table {
}
}
+/// Formats a timestamp string to `yyyyMMddHHmmSSSSS` format.
+///
+/// The function attempts to parse the input timestamp string (`timestamp`) into a `DateTime` object.
+/// It first tries parsing the ISO 8601 format, and if that fails, it tries two other formats: `yyyyMMddHHmmSSSSS`
+/// and `yyyyMMddHHmmSS`. If none of the formats match, it returns the original input string.
+///
+/// # Arguments
+/// - `timestamp`: The input timestamp string to be formatted.
+///
+/// # Returns
+/// A string formatted as `yyyyMMddHHmmSSSSS`. If the input cannot be parsed, the original string is returned.
+fn format_timestamp(timestamp: &str) -> String {
+ if let Ok(datetime) = DateTime::parse_from_rfc3339(timestamp) {
+ return datetime.format("%Y%m%d%H%M%S%3f").to_string();
+ }
+
+ let formats = ["yyyyMMddHHmmSSSSS", "yyyyMMddHHmmSS"];
+ for format in formats.iter() {
+ if let Ok(datetime) = DateTime::parse_from_str(timestamp, format) {
+ return datetime.format("%Y%m%d%H%M%S%3f").to_string();
+ }
+ }
+
+ if timestamp.len() == 10 && timestamp.chars().all(|c| c.is_digit(10) || c == '-') {
Review Comment:
what cases are meant for this check? this checking logic is hard to maintain - please use proper pattern matching to improve the case handling
##########
crates/core/src/table/mod.rs:
##########
@@ -474,6 +481,36 @@ impl Table {
}
}
+/// Formats a timestamp string to `yyyyMMddHHmmSSSSS` format.
+///
+/// The function attempts to parse the input timestamp string (`timestamp`) into a `DateTime` object.
+/// It first tries parsing the ISO 8601 format, and if that fails, it tries two other formats: `yyyyMMddHHmmSSSSS`
+/// and `yyyyMMddHHmmSS`. If none of the formats match, it returns the original input string.
+///
+/// # Arguments
+/// - `timestamp`: The input timestamp string to be formatted.
+///
+/// # Returns
+/// A string formatted as `yyyyMMddHHmmSSSSS`. If the input cannot be parsed, the original string is returned.
+fn format_timestamp(timestamp: &str) -> String {
Review Comment:
i'd expect this returning a Result - if users pass a timestamp str that can't be converted to hudi timeline format properly, then the flow should early return.
The user-provided timestamp should also conform to timezone config of the table - this is by default utc; but it could be configured as `local`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org