You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Blizzara (via GitHub)" <gi...@apache.org> on 2024/08/22 14:10:35 UTC

[PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Blizzara opened a new pull request, #12112:
URL: https://github.com/apache/datafusion/pull/12112

   ## Which issue does this PR close?
   
   ## Rationale for this change
   
   Previously, Substrait didn't have a type & literal for the intervals that combine year/month and day/time, so DF had to use a custom type to support those. https://github.com/substrait-io/substrait/pull/665 however added support for IntervalCompound, which is a superset of DF's IntervalMonthDayNano - so we can switch over.
   
   
   ## What changes are included in this PR?
   
   Add support for consuming Substrait's IntervalCompound type/literals (in addition to existing UDTs, for backwards compatibility)
   Switch Substrait producing to produce IntervalCompound type/literals
   
   ## Are these changes tested?
   
   Tested through existing unit tests.
   
   However, as we now don't produce any UDTs, we cannot easily test the UDT-producing code anymore. 
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   Substrait now produces IntervalCompound types instead of interval-month-day-nano UDTs. Both types can be consumed.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2428055783

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #12112:
URL: https://github.com/apache/datafusion/pull/12112


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2444419400

   Thanks again @Blizzara  @vbarua @westonpace  and @tokoko 👥 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "vbarua (via GitHub)" <gi...@apache.org>.
vbarua commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1819587798


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1831,8 +1832,13 @@ fn from_substrait_type(
                 Ok(DataType::Interval(IntervalUnit::YearMonth))
             }
             r#type::Kind::IntervalDay(_) => Ok(DataType::Interval(IntervalUnit::DayTime)),

Review Comment:
   Possibly out of scope of this PR, but do we need to checking the precision on this type now?
   
   Based on https://github.com/substrait-io/substrait/blob/683f4179a058c2c99c04501b920a48ff372356ff/proto/substrait/type.proto#L132-L140 DataFusion can only handle this type with precision loss when `precision` is set to 0 or 3 (seconds or milliseconds).
   
   
   



##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1842,7 +1848,7 @@ fn from_substrait_type(
                             ),
                     }
                 } else {
-                    // Kept for backwards compatibility, new plans should include the extension instead

Review Comment:
   minor: I would suggest updating this warning here and further down to something like
   ```rust
   // Kept for backwards compatibility
   // Producers should use IntervalCompound instead
   ```
   
   to make it clearer what/who should be using the new type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1820847584


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1842,7 +1848,7 @@ fn from_substrait_type(
                             ),
                     }
                 } else {
-                    // Kept for backwards compatibility, new plans should include the extension instead

Review Comment:
   8c215ea1b16dab457b7c3a107ac61a3338a54425



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818961187


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   That is a very fair question! I think in general the answer is "yes", but there's a catch - the IntervalDayTime in Arrow only supports millisecond precision, so it cannot be done with a single i64::pow. It could be done with two (making it more complicated), or with a fp64::pow (leading to floating point ops). 
   
   But there is an additional question of should we allow the silent precision-losing operations (and the answer is probably no, at least not w/o some user config option) so I think that parts needs a better look anyways and I was thinking of doing it separately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2441293552

   > this should be good to review, fyi @alamb @vbarua @westonpace @tokoko (feel free to lmk if you'd prefer not to be tagged :))
   
   Please do continue to tag me -- thank you very much 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1820627432


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   > No I think you’re right 🙈 math is hard.. I’ll fix this tomorrow. Thanks for catching it!
   
   Marking this PR as draft so we don't accidentally merge it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2444418131

   🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818221896


##########
datafusion/substrait/src/variation_const.rs:
##########
@@ -96,12 +96,16 @@ pub const INTERVAL_DAY_TIME_TYPE_REF: u32 = 2;
 /// [`ScalarValue::IntervalMonthDayNano`]: datafusion::common::ScalarValue::IntervalMonthDayNano
 #[deprecated(
     since = "41.0.0",
-    note = "Use Substrait `UserDefinedType` with name `INTERVAL_MONTH_DAY_NANO_TYPE_NAME` instead"
+    note = "Use Substrait `IntervalCompund` type instead"
 )]
 pub const INTERVAL_MONTH_DAY_NANO_TYPE_REF: u32 = 3;
 
 /// For [`DataType::Interval`] with [`IntervalUnit::MonthDayNano`].
 ///
 /// [`DataType::Interval`]: datafusion::arrow::datatypes::DataType::Interval
 /// [`IntervalUnit::MonthDayNano`]: datafusion::arrow::datatypes::IntervalUnit::MonthDayNano
+#[deprecated(
+    since = "42.1.0",

Review Comment:
   I guess that should be next version, what will it be? :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "tokoko (via GitHub)" <gi...@apache.org>.
tokoko commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818745743


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   Shouldn't precision handling here and on L2294 for IntervalDayToSecond be similar? Seems like only 0, 3, 6, 9 are supported over there. This one makes more sense to me, just curious why they are different.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1820849257


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   fixed in 79f6341266c2042fbb3c1409c14af3e5ffe078a8 and added a test in ea8bc4a780dd7fa917df1d5c36f419f082efa625, thanks again for catching this @westonpace! 
   
   Thanks @alamb, I  converted back to PR now and I think it's ready to merge :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2444299061

   I'll plan to merge this PR once the CI checks have completed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1819888903


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   I'm probably butchering this but should this be `9 - *p` instead of `*p - 9`?  If precision is 0 then we want 10^9 right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818222135


##########
datafusion/substrait/src/logical_plan/producer.rs:
##########
@@ -2310,39 +2306,6 @@ mod test {
         Ok(())
     }
 
-    #[test]
-    fn custom_type_literal_extensions() -> Result<()> {

Review Comment:
   sadly as we no longer produce custom types, there's no easy way to test for them either :/ 
   maybe if we one day implement some UDT support then that can be used for testing here instead. But for now there's no good replacement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#issuecomment-2440329051

   this should be good to review, fyi @alamb @vbarua @westonpace @tokoko (feel free to lmk if you'd prefer not to be tagged :))


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1820849833


##########
datafusion/substrait/src/variation_const.rs:
##########
@@ -96,12 +96,16 @@ pub const INTERVAL_DAY_TIME_TYPE_REF: u32 = 2;
 /// [`ScalarValue::IntervalMonthDayNano`]: datafusion::common::ScalarValue::IntervalMonthDayNano
 #[deprecated(
     since = "41.0.0",
-    note = "Use Substrait `UserDefinedType` with name `INTERVAL_MONTH_DAY_NANO_TYPE_NAME` instead"
+    note = "Use Substrait `IntervalCompund` type instead"
 )]
 pub const INTERVAL_MONTH_DAY_NANO_TYPE_REF: u32 = 3;
 
 /// For [`DataType::Interval`] with [`IntervalUnit::MonthDayNano`].
 ///
 /// [`DataType::Interval`]: datafusion::arrow::datatypes::DataType::Interval
 /// [`IntervalUnit::MonthDayNano`]: datafusion::arrow::datatypes::IntervalUnit::MonthDayNano
+#[deprecated(
+    since = "42.1.0",

Review Comment:
   updated in d527b4ec1d83874fd04bf76c4a96da53c164978e



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818879812


##########
datafusion/substrait/src/variation_const.rs:
##########
@@ -96,12 +96,16 @@ pub const INTERVAL_DAY_TIME_TYPE_REF: u32 = 2;
 /// [`ScalarValue::IntervalMonthDayNano`]: datafusion::common::ScalarValue::IntervalMonthDayNano
 #[deprecated(
     since = "41.0.0",
-    note = "Use Substrait `UserDefinedType` with name `INTERVAL_MONTH_DAY_NANO_TYPE_NAME` instead"
+    note = "Use Substrait `IntervalCompund` type instead"
 )]
 pub const INTERVAL_MONTH_DAY_NANO_TYPE_REF: u32 = 3;
 
 /// For [`DataType::Interval`] with [`IntervalUnit::MonthDayNano`].
 ///
 /// [`DataType::Interval`]: datafusion::arrow::datatypes::DataType::Interval
 /// [`IntervalUnit::MonthDayNano`]: datafusion::arrow::datatypes::IntervalUnit::MonthDayNano
+#[deprecated(
+    since = "42.1.0",

Review Comment:
   43 I believe: https://github.com/apache/datafusion/issues/12470



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1819955199


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   No I think you’re right 🙈 math is hard.. I’ll fix this tomorrow. Thanks for catching it!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "Blizzara (via GitHub)" <gi...@apache.org>.
Blizzara commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1819756919


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -1831,8 +1832,13 @@ fn from_substrait_type(
                 Ok(DataType::Interval(IntervalUnit::YearMonth))
             }
             r#type::Kind::IntervalDay(_) => Ok(DataType::Interval(IntervalUnit::DayTime)),

Review Comment:
   Yup see https://github.com/apache/datafusion/pull/12112#discussion_r1818961187. I think we should change it, maybe with some option so that user can decide if they want to throw or accept the precision loss. But will be a separate PR :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1820627432


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   > No I think you’re right 🙈 math is hard.. I’ll fix this tomorrow. Thanks for catching it!
   
   Marking this PR as draft so we don't accidentally merge it until it is fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Support Substrait's IntervalCompound type/literal instead of interval-month-day-nano UDT [datafusion]

Posted by "tokoko (via GitHub)" <gi...@apache.org>.
tokoko commented on code in PR #12112:
URL: https://github.com/apache/datafusion/pull/12112#discussion_r1818971965


##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -2299,6 +2306,35 @@ fn from_substrait_literal(
         Some(LiteralType::IntervalYearToMonth(IntervalYearToMonth { years, months })) => {
             ScalarValue::new_interval_ym(*years, *months)
         }
+        Some(LiteralType::IntervalCompound(IntervalCompound {
+            interval_year_to_month,
+            interval_day_to_second,
+        })) => match (interval_year_to_month, interval_day_to_second) {
+            (
+                Some(IntervalYearToMonth { years, months }),
+                Some(IntervalDayToSecond {
+                    days,
+                    seconds,
+                    subseconds,
+                    precision_mode:
+                        Some(interval_day_to_second::PrecisionMode::Precision(p)),
+                }),
+            ) => {
+                if *p < 0 || *p > 9 {
+                    return plan_err!(
+                        "Unsupported Substrait interval day to second precision: {}",
+                        p
+                    );
+                }
+                let nanos = *subseconds * i64::pow(10, (*p - 9) as u32);

Review Comment:
   sounds good. I think there are two questions (probably unrelated?) to explore here: 1) precision-losing (for p>3) and 2) whether in-between values are valid (p==1, p==2)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org