You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "dami0806 (via GitHub)" <gi...@apache.org> on 2025/10/12 10:31:21 UTC
[PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
dami0806 opened a new pull request, #6650:
URL: https://github.com/apache/opendal/pull/6650
# Related Issue
Closes #6647
# Rationale
This PR centralizes all date/time-related logic within OpenDAL. By introducing a new `Timestamp` wrapper in `core/src/raw/time.rs`, direct usage of `jiff::Timestamp` throughout the codebase is removed.
**Key benefits:**
- **Simplified API**: Provides convenience methods like `now()`, `from_millisecond()`, and `format_http_date()`.
- **Consistent error handling**: Converts `jiff` errors into OpenDAL's `Error` type.
- **Type safety**: Prevents external `jiff` types from leaking into the public API.
- **Future-proof design**: Makes it easier to adapt to potential date/time library changes.
---
# Changes included in this PR
This PR consists of **8 commits**, clearly separated for easy review:
### 1. Add `Timestamp` wrapper (`time.rs`)
- Introduced a new `Timestamp` wrapper around `jiff::Timestamp` in `core/src/raw/time.rs`.
- Implemented conversion traits: `From`, `TryFrom`, `FromStr`.
- Added formatting methods: `format_http_date`, `format_rfc3339`, `format_iso8601`.
- Added helper constructors: `now`, `from_millisecond`, `from_second`.
### 2. Remove `jiff_util` module
- Deleted the obsolete `core/src/raw/jiff_util.rs`.
- Updated `core/src/raw/mod.rs` to expose the new `Timestamp` type.
### 3. Update raw layer utilities
- Refactored `adapters`, `http_util`, and `ops` to use the `Timestamp` wrapper.
### 4. Update types layer
- Updated `metadata`, `options`, and `operator` to use the `Timestamp` wrapper.
### 5. Migrate cloud storage services
- Services: S3, GCS, Azure Blob/DLS/File, Tencent COS, Alibaba OSS, Backblaze B2, Aliyun Drive, Vercel Blob, Cloudflare KV.
- Metadata parsing and formatting now use the `Timestamp` wrapper.
### 6. Migrate filesystem services
- Services: Local FS, HDFS, HDFS Native, SFTP, WebHDFS, WebDAV, FTP, MonoioFS, Alluxio, CompFS.
- Updated file metadata, system time conversions, and listers/writers to use `Timestamp`.
### 7. Migrate other services
- Services: Dropbox, OneDrive, Google Drive, pCloud, GitHub, Hugging Face, Koofr, LakeFS, Seafile, Yandex Disk, UpYun, Swift, cacache, dashmap, mini_moka, DBFS.
- Integrated the `Timestamp` wrapper.
### 8. Update `Cargo.lock`
- Refreshed dependency lockfile to reflect the `Timestamp` migration.
---
### Helper function replacements
Throughout the migration:
- Replaced `parse_datetime_from_rfc3339()` → `str.parse::<Timestamp>()`.
- Replaced `parse_datetime_from_timestamp_millis()` → `Timestamp::from_millisecond()`.
- Replaced direct `jiff::Timestamp` usage → `Timestamp` wrapper.
---
# Commit Summary
| # | Commit ID | Description |
|---|-----------|-------------|
| 1 | f672ba0a | Add `Timestamp` wrapper (`time.rs`) |
| 2 | d3e90f00 | Remove `jiff_util` module |
| 3 | 9e045d4a | Update raw utilities |
| 4 | 44267e25 | Update types layer |
| 5 | 3b5a2f99 | Migrate cloud storage services |
| 6 | c5242809 | Migrate filesystem services |
| 7 | 8ebbb36a | Migrate other services |
| 8 | 96407992 | Update `Cargo.lock` |
---
# Local Verification
- ✅ `cargo fmt --check`
- ✅ `cargo check -p opendal --features="blocking"`
- ✅ `cargo clippy -p opendal --features="blocking" -- -D warnings`
- ✅ `cargo test -p opendal --features="blocking,services-s3,services-fs"` (All 319 tests passed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2427972406
##########
bindings/java/src/convert.rs:
##########
@@ -137,14 +136,17 @@ pub(crate) fn read_instant_field_to_timestamp(
.call_method(&result, "getEpochSecond", "()J", &[])?
.j()?;
let nano = env.call_method(&result, "getNano", "()I", &[])?.i()?;
- Timestamp::new(epoch_second, nano).map(Some).map_err(|err| {
- Error::new(
- ErrorKind::Unexpected,
- format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
- )
- .set_source(err)
- .into()
- })
+
+ opendal::raw::Timestamp::from_second(epoch_second)
+ .map(Some)
+ .map_err(|err| {
+ Error::new(
+ ErrorKind::Unexpected,
+ format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
+ )
+ .set_source(err)
+ .into()
+ })
}
Review Comment:
Here is a bug I'll fix then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "dami0806 (via GitHub)" <gi...@apache.org>.
dami0806 commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2423805720
##########
bindings/c/src/metadata.rs:
##########
@@ -131,7 +131,9 @@ impl opendal_metadata {
let mtime = self.deref().last_modified();
match mtime {
None => -1,
- Some(time) => time.as_millisecond(),
+ Some(time) => {
+ time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
+ }
Review Comment:
While updating multiple bindings at once, I missed the temporary workaround I had added earlier for millisecond calculation and forgot to revert it... I’m sorry for the confusion this caused during review.
I will fix this immediately as follows:
Add a pub fn as_millisecond() method to the opendal::raw::Timestamp wrapper.
Replace the temporary calculation in the C binding with time.as_millisecond() for clarity and consistency.
I’ll be more careful and thorough in my future reviews and testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "dami0806 (via GitHub)" <gi...@apache.org>.
dami0806 commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2424022715
##########
bindings/c/src/metadata.rs:
##########
@@ -131,7 +131,9 @@ impl opendal_metadata {
let mtime = self.deref().last_modified();
match mtime {
None => -1,
- Some(time) => time.as_millisecond(),
+ Some(time) => {
+ time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
+ }
Review Comment:
When I created the Timestamp wrapper, I added methods like as_second() and from_millisecond(), but I forgot to add as_millisecond().
I had implemented as_second() and from_millisecond() in the Timestamp wrapper, but as_millisecond() was missing.
While implementing the C bindings, this caused a compilation error because the method didn’t exist.
To quickly unblock myself and continue testing other parts, I used a temporary workaround with manual calculation (as_second() * 1000 + subsec_nanosecond() / 1_000_000).
After the tests passed, I should have gone back, properly added as_millisecond() to the Timestamp wrapper, and cleaned up the temporary code, but I missed it.
I should have been more careful and ensured that temporary workarounds were properly cleaned up before committing. I apologize for leaving this temporary code in the final commit.
I will now fix this properly by adding as_millisecond() to the Timestamp wrapper in core/src/raw/time.rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2427951802
##########
bindings/python/src/metadata.rs:
##########
@@ -121,8 +120,8 @@ impl Metadata {
/// Last modified time
#[getter]
- pub fn last_modified(&self) -> Option<Timestamp> {
- self.0.last_modified()
+ pub fn last_modified(&self) -> Option<jiff::Timestamp> {
Review Comment:
Cool, `jiff::Timestamp` has native pyo3 support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "dami0806 (via GitHub)" <gi...@apache.org>.
dami0806 commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2423805720
##########
bindings/c/src/metadata.rs:
##########
@@ -131,7 +131,9 @@ impl opendal_metadata {
let mtime = self.deref().last_modified();
match mtime {
None => -1,
- Some(time) => time.as_millisecond(),
+ Some(time) => {
+ time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
+ }
Review Comment:
While updating multiple bindings at once, I missed the temporary workaround I had added earlier for millisecond calculation and forgot to revert it... I’m sorry for the confusion this caused during review.
I will fix this immediately as follows:
Add a pub fn as_millisecond() method to the opendal::raw::Timestamp wrapper.
Replace the temporary calculation in the C binding with time.as_millisecond() for clarity and consistency.
I’ll be more careful and thorough in my future reviews and testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2423721593
##########
bindings/c/src/metadata.rs:
##########
@@ -131,7 +131,9 @@ impl opendal_metadata {
let mtime = self.deref().last_modified();
match mtime {
None => -1,
- Some(time) => time.as_millisecond(),
+ Some(time) => {
+ time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
+ }
Review Comment:
This seems like something vibe coding produced.
Please try to review your code to make it more concise for human review. Otherwise I don't think it's valuable for requesting for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #6650:
URL: https://github.com/apache/opendal/pull/6650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2428013322
##########
bindings/java/src/convert.rs:
##########
@@ -137,14 +136,17 @@ pub(crate) fn read_instant_field_to_timestamp(
.call_method(&result, "getEpochSecond", "()J", &[])?
.j()?;
let nano = env.call_method(&result, "getNano", "()I", &[])?.i()?;
- Timestamp::new(epoch_second, nano).map(Some).map_err(|err| {
- Error::new(
- ErrorKind::Unexpected,
- format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
- )
- .set_source(err)
- .into()
- })
+
+ opendal::raw::Timestamp::from_second(epoch_second)
+ .map(Some)
+ .map_err(|err| {
+ Error::new(
+ ErrorKind::Unexpected,
+ format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
+ )
+ .set_source(err)
+ .into()
+ })
}
Review Comment:
Follow up at https://github.com/apache/opendal/pull/6663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "dami0806 (via GitHub)" <gi...@apache.org>.
dami0806 commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2424022715
##########
bindings/c/src/metadata.rs:
##########
@@ -131,7 +131,9 @@ impl opendal_metadata {
let mtime = self.deref().last_modified();
match mtime {
None => -1,
- Some(time) => time.as_millisecond(),
+ Some(time) => {
+ time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
+ }
Review Comment:
When I implemented the Timestamp wrapper, I added methods like as_second() and from_millisecond(), but initially missed as_millisecond().
While working on the C bindings, this caused a compilation error. To unblock testing, I temporarily calculated milliseconds manually (as_second() * 1000 + subsec_nanosecond() / 1_000_000).
I should have cleaned up this temporary code afterward. I’ll now properly add as_millisecond() to the Timestamp wrapper in core/src/raw/time.rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Centralize date/time handling with Timestamp wrapper [opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #6650:
URL: https://github.com/apache/opendal/pull/6650#discussion_r2427970732
##########
bindings/python/src/metadata.rs:
##########
@@ -121,8 +120,8 @@ impl Metadata {
/// Last modified time
#[getter]
- pub fn last_modified(&self) -> Option<Timestamp> {
- self.0.last_modified()
+ pub fn last_modified(&self) -> Option<jiff::Timestamp> {
Review Comment:
Yes.
https://github.com/PyO3/pyo3/pull/4823 and others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org