You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "simonvandel (via GitHub)" <gi...@apache.org> on 2023/11/22 16:55:47 UTC

[PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

simonvandel opened a new pull request, #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308

   
   
   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #8307.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   Allow other default values than Int64 on lag/lead window functions.
   
   ## What changes are included in this PR?
   
   If a default value is specified, it is now coerced if possible.
   Before this PR only Int64 was allowed.
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   Yes, added some slt tests.
   
   ## Are there any user-facing changes?
   
   Yes, more default values are now accepted.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412113005


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I was suggesting `comparison_coercion` with comment.
   
   But, I rethink about it again, now I think  
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   is better for here.
   
   btw, i dont think your comment `// We explicitely allow a lossy conversion between signed and unsigned integers` is correct 🤔 



##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I was suggesting `comparison_coercion` with comment.
   
   But, I rethink about it again, now I think  
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   is better for here.
   
   btw, i dont think your comment `// We explicitely allow a lossy conversion between signed and unsigned integers` is correct 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412140136


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I suggest `comparison_coercion`. I read the doc, it does not seem lag/lead should only be integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412144585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Note that coerced_type might not be the same of `dtype` or `default_value_type`. So we need to cast both of them to  `coerced_type`
   ```
   if let(coerced_type) = comparison_coercion(dtype, &default_value_type) {}
   ```
   
   For example, if we have (dtype: UInt64, default_value_type: Int32), coerced_type is Int64



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412522516


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I think coercing during physical evaluation is too late in the process -- most other coercion happens in the analyze phase so that subsequent transformations / optimizations use the same rules.  
   
   Would it be possible to move this coercion earlier into 
   
   https://github.com/apache/arrow-datafusion/blob/513fd052bdbf5c7a73de544a876961f780b90a92/datafusion/optimizer/src/analyzer/type_coercion.rs#L64-L63
   
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412149169


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   cc @alamb to confirm my suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1410327873


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -235,11 +236,27 @@ fn get_default_value(
     default_value: Option<&ScalarValue>,
     dtype: &DataType,
 ) -> Result<ScalarValue> {
-    if let Some(value) = default_value {
-        if let ScalarValue::Int64(Some(val)) = value {
-            ScalarValue::try_from_string(val.to_string(), dtype)
+    if let Some(default_value) = default_value {
+        let default_value_type = default_value.data_type();
+        if can_coerce_from(dtype, &default_value_type)
+            || (dtype.is_integer() && default_value_type.is_integer())
+        {
+            ScalarValue::try_from_array(
+                &cast_with_options(
+                    &default_value.to_array()?,
+                    dtype,
+                    &CastOptions {
+                        safe: false,
+                        format_options: Default::default(),
+                    },
+                )?,
+                0,
+            )
         } else {
-            internal_err!("Expects default value to have Int64 type")
+            Err(DataFusionError::Internal(format!(

Review Comment:
   Yes, thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1410744262


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Sorry, I didn't understand what you mean by "change integer". Could you rephrase, or provide a suggestion for a change? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1410549829


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   It seems that you just need to change integer, since I'm not sure having `pub fn comparison_binary_numeric_coercion` is a good idea or not. At least, we can add the comment for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1411421025


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Code from your previous commit
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   
   Comment similar to
   // We only check coercion between integer.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1402410235


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -235,11 +236,27 @@ fn get_default_value(
     default_value: Option<&ScalarValue>,
     dtype: &DataType,
 ) -> Result<ScalarValue> {
-    if let Some(value) = default_value {
-        if let ScalarValue::Int64(Some(val)) = value {
-            ScalarValue::try_from_string(val.to_string(), dtype)
+    if let Some(default_value) = default_value {
+        let default_value_type = default_value.data_type();
+        if can_coerce_from(dtype, &default_value_type)
+            || (dtype.is_integer() && default_value_type.is_integer())

Review Comment:
   I added this because otherwise existing tests would fail coercing between UInt64 and Int64 which would regress functionality



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1411421025


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   // We only check coercion between integer.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412144585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   ```
   if let(coerced_type) = comparison_coercion(dtype, &default_value_type) {}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1410328591


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -235,11 +236,27 @@ fn get_default_value(
     default_value: Option<&ScalarValue>,
     dtype: &DataType,
 ) -> Result<ScalarValue> {
-    if let Some(value) = default_value {
-        if let ScalarValue::Int64(Some(val)) = value {
-            ScalarValue::try_from_string(val.to_string(), dtype)
+    if let Some(default_value) = default_value {
+        let default_value_type = default_value.data_type();
+        if can_coerce_from(dtype, &default_value_type)
+            || (dtype.is_integer() && default_value_type.is_integer())

Review Comment:
   `comparison_binary_numeric_coercion` is not public, but I changed to `comparison_coercion` which seems to work fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1408589220


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -235,11 +236,27 @@ fn get_default_value(
     default_value: Option<&ScalarValue>,
     dtype: &DataType,
 ) -> Result<ScalarValue> {
-    if let Some(value) = default_value {
-        if let ScalarValue::Int64(Some(val)) = value {
-            ScalarValue::try_from_string(val.to_string(), dtype)
+    if let Some(default_value) = default_value {
+        let default_value_type = default_value.data_type();
+        if can_coerce_from(dtype, &default_value_type)
+            || (dtype.is_integer() && default_value_type.is_integer())
+        {
+            ScalarValue::try_from_array(
+                &cast_with_options(
+                    &default_value.to_array()?,
+                    dtype,
+                    &CastOptions {
+                        safe: false,
+                        format_options: Default::default(),
+                    },
+                )?,
+                0,
+            )
         } else {
-            internal_err!("Expects default value to have Int64 type")
+            Err(DataFusionError::Internal(format!(

Review Comment:
   Can we use macro here? See https://github.com/apache/arrow-datafusion/issues/7360



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#issuecomment-1836701890

   (p.s sorry for the late review)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#issuecomment-1858375343

   Marking as draft as i think this is no longer waiting on feedback. Please mark it ready for review if/when it is ready for another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1411940640


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   So just to be clear, are you suggesting that the comparison should be reverted to 
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               // We explicitely allow a lossy conversion between signed and unsigned integers
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412113005


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I was suggesting `comparison_coercion` with comment actually
   
   But, I rethink about it again, now I think  
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   is better for here.
   
   btw, i dont think your comment `// We explicitely allow a lossy conversion between signed and unsigned integers` is correct 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1408601940


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -235,11 +236,27 @@ fn get_default_value(
     default_value: Option<&ScalarValue>,
     dtype: &DataType,
 ) -> Result<ScalarValue> {
-    if let Some(value) = default_value {
-        if let ScalarValue::Int64(Some(val)) = value {
-            ScalarValue::try_from_string(val.to_string(), dtype)
+    if let Some(default_value) = default_value {
+        let default_value_type = default_value.data_type();
+        if can_coerce_from(dtype, &default_value_type)
+            || (dtype.is_integer() && default_value_type.is_integer())

Review Comment:
   Is `comparison_binary_numeric_coercion` possible for your case?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412113005


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I was suggesting `comparison_coercion` with comment actually
   
   But, I rethink about it again, now I think  
   ```rust
   if can_coerce_from(dtype, &default_value_type)
               || (dtype.is_integer() && default_value_type.is_integer())
   ```
   is better for here.
   
   btw, i dont think your comment `// We explicitely allow a lossy conversion between signed and unsigned integers` is correct 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412124585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I thought you were just trying to deal with integer type. But, I think we need to cover any compatible type 🤔 ?
   
   If that is true, we should do `if let (coerced_type) = comparison_coercion(&default_value_type, dtype)` and cast to coerced_type. Note that coerced_type might not be the same as `default_value_type` or `dtype`.
   
   If we just need to convert dtype to default_value_type, then `can_coerce_from` is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412144585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Note that coerced_type might not be the same of `dtype` or `default_value_type`. So we need to cast both of them to  `coerced_type`
   ```
   if let(coerced_type) = comparison_coercion(dtype, &default_value_type) {}
   ```
   
   For example, if we have (UInt64, Int32), coerced_type is Int64



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412144585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Note that coerced_type might not be the same of `dtype` or `default_value_type`. So we need to cast both of them to  `coerced_type`
   ```
   if let(coerced_type) = comparison_coercion(dtype, &default_value_type) {}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1411417986


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   Sorry, I mean coerce, not change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412124585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I thought you were just trying to deal with integer type. But, I think we need to cover any compatible type 🤔 ?
   
   If that is true, we can stick with `if comparison_coercion(&default_value_type, dtype).is_some()`



##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I thought you were just trying to deal with integer type. But, I think we need to cover any compatible type 🤔 ?
   
   If that is true, we can stick with `if comparison_coercion(&default_value_type, dtype).is_some()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #8308:
URL: https://github.com/apache/arrow-datafusion/pull/8308#discussion_r1412124585


##########
datafusion/physical-expr/src/window/lead_lag.rs:
##########
@@ -238,9 +238,8 @@ fn get_default_value(
 ) -> Result<ScalarValue> {
     if let Some(default_value) = default_value {
         let default_value_type = default_value.data_type();
-        if can_coerce_from(dtype, &default_value_type)
-            || (dtype.is_integer() && default_value_type.is_integer())
-        {
+
+        if comparison_coercion(&default_value_type, dtype).is_some() {

Review Comment:
   I thought you were just trying to deal with integer type. But, I think we need to cover any compatible type 🤔 ?
   
   If that is true, we should do `if let (coerced_type) = comparison_coercion(&default_value_type, dtype)` and cast to coerced_type. Note that coerced_type might not be the same as `default_value_type` or `dtype`.
   
   If we just need to convert dtype to default_value_type, then `can_coerce_from` is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow default value on lag/lead if they are coerceable to value [arrow-datafusion]

Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel closed pull request #8308: Allow default value on lag/lead if they are coerceable to value
URL: https://github.com/apache/arrow-datafusion/pull/8308


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org