You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mayank172900 (via GitHub)" <gi...@apache.org> on 2026/08/02 07:31:33 UTC
[PR] Support casting strings to durations [arrow-rs]
mayank172900 opened a new pull request, #10516:
URL: https://github.com/apache/arrow-rs/pull/10516
# Which issue does this PR close?
- Closes #4555.
# Rationale for this change
`arrow-cast` can format durations as strings but does not currently support casting those strings back to duration arrays. This leaves duration casts asymmetric and prevents round-tripping formatted values.
# What changes are included in this PR?
- Allow `Utf8`, `LargeUtf8`, and `Utf8View` values to cast to all four duration units.
- Parse human-readable interval-style durations and the ISO 8601 representation emitted by the duration formatter.
- Treat unitless integers as values in the target duration unit.
- Reject year and month fields because they do not have fixed durations.
- Preserve safe-cast behavior by converting invalid inputs to nulls and returning errors for unsafe casts.
# Are these changes tested?
Yes. Tests cover all duration units, all supported string representations, safe and unsafe behavior, invalid calendar units, and ISO 8601 and pretty-format round trips.
Validation performed:
- `cargo test -p arrow-cast --lib` (363 passed)
- `cargo clippy -p arrow-cast --lib --tests -- -D warnings`
- `cargo fmt --all -- --check`
# Are there any user-facing changes?
Yes. Users can now cast string arrays to duration arrays. This adds functionality without changing an existing public API.
# Tooling disclosure
Generative tooling assisted with the parser implementation, cast dispatch arms, tests, and PR wording. The full patch was locally reviewed and verified with the checks above. This draft should be marked ready only after the contributor has reviewed every detail and is prepared to own and debug the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support casting strings to durations [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10516: Support casting strings to durations
URL: https://github.com/apache/arrow-rs/pull/10516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support casting strings to durations [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10516:
URL: https://github.com/apache/arrow-rs/pull/10516#discussion_r3709602995
##########
arrow-cast/src/parse.rs:
##########
@@ -1090,11 +1090,82 @@ pub fn parse_interval_month_day_nano(
parse_interval_month_day_nano_config(value, IntervalParseConfig::new(IntervalUnit::Month))
}
+/// Parse a human-readable or ISO 8601 duration string to an Arrow duration value.
+///
+/// Human-readable durations use the same syntax as intervals, for example
+/// `2 days 3 hours 4.5 seconds`. Values without an explicit unit use the unit
+/// of `T`. Year and month fields are rejected because their lengths are not
+/// fixed. ISO 8601 strings produced by [`crate::display::DurationFormat::ISO8601`]
+/// are also supported.
+pub(crate) fn parse_duration<T: ArrowTemporalType<Native = i64>>(
+ value: &str,
+) -> Result<i64, ArrowError> {
+ let (default_unit, scale) = match T::DATA_TYPE {
+ DataType::Duration(TimeUnit::Second) => (IntervalUnit::Second, NANOS_PER_SECOND),
+ DataType::Duration(TimeUnit::Millisecond) => (IntervalUnit::Millisecond, NANOS_PER_MILLIS),
+ DataType::Duration(TimeUnit::Microsecond) => (IntervalUnit::Microsecond, 1_000),
+ DataType::Duration(TimeUnit::Nanosecond) => (IntervalUnit::Nanosecond, 1),
+ _ => unreachable!(),
+ };
+
+ let value = value.trim_ascii();
+
+ // Preserve the full i64 range for the common case of a unitless integer.
+ if let Ok(value) = value.parse::<i64>() {
+ return Ok(value);
+ }
+
+ // Duration display currently emits ISO 8601 values as a number of seconds,
+ // for example `PT1.5S` or `-PT1.5S`. Convert this to the interval parser's
+ // human-readable syntax so both representations share the same validation.
+ let normalized;
+ let value = if let Some(seconds) = value
+ .strip_prefix("PT")
+ .and_then(|value| value.strip_suffix('S'))
+ {
+ normalized = format!("{seconds} seconds");
+ normalized.as_str()
+ } else if let Some(seconds) = value
+ .strip_prefix("-PT")
+ .and_then(|value| value.strip_suffix('S'))
+ {
+ normalized = format!("-{seconds} seconds");
+ normalized.as_str()
+ } else {
+ value
+ };
Review Comment:
Do we need to consider other designators such as hours, minutes, etc.?
also is this possible to parse directly without reconstructing a string to pass through the interval parser?
##########
arrow-cast/src/parse.rs:
##########
@@ -1090,11 +1090,82 @@ pub fn parse_interval_month_day_nano(
parse_interval_month_day_nano_config(value, IntervalParseConfig::new(IntervalUnit::Month))
}
+/// Parse a human-readable or ISO 8601 duration string to an Arrow duration value.
+///
+/// Human-readable durations use the same syntax as intervals, for example
+/// `2 days 3 hours 4.5 seconds`. Values without an explicit unit use the unit
+/// of `T`. Year and month fields are rejected because their lengths are not
+/// fixed. ISO 8601 strings produced by [`crate::display::DurationFormat::ISO8601`]
+/// are also supported.
+pub(crate) fn parse_duration<T: ArrowTemporalType<Native = i64>>(
+ value: &str,
+) -> Result<i64, ArrowError> {
+ let (default_unit, scale) = match T::DATA_TYPE {
+ DataType::Duration(TimeUnit::Second) => (IntervalUnit::Second, NANOS_PER_SECOND),
+ DataType::Duration(TimeUnit::Millisecond) => (IntervalUnit::Millisecond, NANOS_PER_MILLIS),
+ DataType::Duration(TimeUnit::Microsecond) => (IntervalUnit::Microsecond, 1_000),
+ DataType::Duration(TimeUnit::Nanosecond) => (IntervalUnit::Nanosecond, 1),
+ _ => unreachable!(),
+ };
+
+ let value = value.trim_ascii();
+
+ // Preserve the full i64 range for the common case of a unitless integer.
+ if let Ok(value) = value.parse::<i64>() {
+ return Ok(value);
+ }
+
+ // Duration display currently emits ISO 8601 values as a number of seconds,
+ // for example `PT1.5S` or `-PT1.5S`. Convert this to the interval parser's
+ // human-readable syntax so both representations share the same validation.
+ let normalized;
+ let value = if let Some(seconds) = value
+ .strip_prefix("PT")
+ .and_then(|value| value.strip_suffix('S'))
+ {
+ normalized = format!("{seconds} seconds");
+ normalized.as_str()
+ } else if let Some(seconds) = value
+ .strip_prefix("-PT")
+ .and_then(|value| value.strip_suffix('S'))
+ {
+ normalized = format!("-{seconds} seconds");
+ normalized.as_str()
+ } else {
+ value
+ };
+
+ let config = IntervalParseConfig::new(default_unit);
+ let components = parse_interval_components(value, &config)?;
+
+ if components.iter().any(|(_, unit)| {
+ matches!(
+ unit,
+ IntervalUnit::Century | IntervalUnit::Decade | IntervalUnit::Year | IntervalUnit::Month
+ )
+ }) {
+ return Err(ArrowError::CastError(format!(
+ "Cannot cast {value} to {}. Year and month fields are not supported.",
Review Comment:
probably need the error message to be more generic here, as it disallows century, decade, etc.
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -5896,6 +5933,152 @@ mod tests {
);
}
+ #[test]
+ fn test_cast_string_to_duration() {
+ let source = vec![
+ Some("2"),
+ Some("1.5 seconds"),
+ Some("2 minutes"),
+ Some("1 day"),
+ Some("PT0.000001S"),
+ Some("-PT0.000001S"),
+ Some("1 month"),
+ Some("foobar"),
+ None,
+ ];
+
+ macro_rules! test_duration {
Review Comment:
this macro can probably be a generic
##########
arrow-cast/src/parse.rs:
##########
@@ -1090,11 +1090,82 @@ pub fn parse_interval_month_day_nano(
parse_interval_month_day_nano_config(value, IntervalParseConfig::new(IntervalUnit::Month))
}
+/// Parse a human-readable or ISO 8601 duration string to an Arrow duration value.
+///
+/// Human-readable durations use the same syntax as intervals, for example
+/// `2 days 3 hours 4.5 seconds`. Values without an explicit unit use the unit
+/// of `T`. Year and month fields are rejected because their lengths are not
+/// fixed. ISO 8601 strings produced by [`crate::display::DurationFormat::ISO8601`]
+/// are also supported.
+pub(crate) fn parse_duration<T: ArrowTemporalType<Native = i64>>(
+ value: &str,
+) -> Result<i64, ArrowError> {
+ let (default_unit, scale) = match T::DATA_TYPE {
+ DataType::Duration(TimeUnit::Second) => (IntervalUnit::Second, NANOS_PER_SECOND),
+ DataType::Duration(TimeUnit::Millisecond) => (IntervalUnit::Millisecond, NANOS_PER_MILLIS),
+ DataType::Duration(TimeUnit::Microsecond) => (IntervalUnit::Microsecond, 1_000),
+ DataType::Duration(TimeUnit::Nanosecond) => (IntervalUnit::Nanosecond, 1),
+ _ => unreachable!(),
+ };
+
+ let value = value.trim_ascii();
+
+ // Preserve the full i64 range for the common case of a unitless integer.
+ if let Ok(value) = value.parse::<i64>() {
+ return Ok(value);
+ }
Review Comment:
i'm not sure about this behaviour as it does seem a bit implicit 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support casting strings to durations [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10516:
URL: https://github.com/apache/arrow-rs/pull/10516#issuecomment-6072761105
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org