You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/13 04:43:16 UTC

[GitHub] [arrow-datafusion] liukun4515 opened a new pull request #1554: support mathematics operation for decimal data type

liukun4515 opened a new pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554


   # Which issue does this PR close?
    
   part of #122 
   
   
    # Rationale for this change
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   # What changes are included in this PR?
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are there any user-facing changes?
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786079490



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786079635



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {
+    match numeric_type {
+        DataType::Int8 => Some(DataType::Decimal(3, 0)),
+        DataType::Int16 => Some(DataType::Decimal(5, 0)),
+        DataType::Int32 => Some(DataType::Decimal(10, 0)),
+        DataType::Int64 => Some(DataType::Decimal(20, 0)),
+        // TODO if we convert the floating-point data to the decimal type, it maybe overflow.
+        DataType::Float32 => Some(DataType::Decimal(14, 7)),
+        DataType::Float64 => Some(DataType::Decimal(30, 15)),
+        _ => None,
+    }
+}
+
+fn mathematics_numerical_coercion(

Review comment:
       ![image](https://user-images.githubusercontent.com/7450163/149791191-ed11a0f5-8794-44b7-a695-b83b82e9fb3c.png)
   The `numerical_coercion` is used by other coercion functions:
   1. `numerical_coercion`->`eq_coercion`  -> `Operator::IsDistinctFrom | Operator::IsNotDistinctFrom => `
   2. `numerical_coercion` -> `dictionary_value_coercion`
   If I extend the `numerical_coercion`, it may take a side effect to other coercion logical.
   I will refactor and merge these two together and make it clear if there is no conflict.
   @alamb 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786495477



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {
+    match numeric_type {
+        DataType::Int8 => Some(DataType::Decimal(3, 0)),
+        DataType::Int16 => Some(DataType::Decimal(5, 0)),
+        DataType::Int32 => Some(DataType::Decimal(10, 0)),
+        DataType::Int64 => Some(DataType::Decimal(20, 0)),
+        // TODO if we convert the floating-point data to the decimal type, it maybe overflow.
+        DataType::Float32 => Some(DataType::Decimal(14, 7)),
+        DataType::Float64 => Some(DataType::Decimal(30, 15)),
+        _ => None,
+    }
+}
+
+fn mathematics_numerical_coercion(

Review comment:
       @alamb  I will extend it in the follow-up pull request and support dict and `distinct` with decimal.
   In the follow-up pull request, I think we can refactor out a common part.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786108285



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {
+    match numeric_type {
+        DataType::Int8 => Some(DataType::Decimal(3, 0)),
+        DataType::Int16 => Some(DataType::Decimal(5, 0)),
+        DataType::Int32 => Some(DataType::Decimal(10, 0)),
+        DataType::Int64 => Some(DataType::Decimal(20, 0)),
+        // TODO if we convert the floating-point data to the decimal type, it maybe overflow.
+        DataType::Float32 => Some(DataType::Decimal(14, 7)),
+        DataType::Float64 => Some(DataType::Decimal(30, 15)),
+        _ => None,
+    }
+}
+
+fn mathematics_numerical_coercion(

Review comment:
       did you try extending `numerical_coercion` (and if so, did it have any side effects)?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb merged pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786099180



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {
+    match numeric_type {
+        DataType::Int8 => Some(DataType::Decimal(3, 0)),
+        DataType::Int16 => Some(DataType::Decimal(5, 0)),
+        DataType::Int32 => Some(DataType::Decimal(10, 0)),
+        DataType::Int64 => Some(DataType::Decimal(20, 0)),
+        // TODO if we convert the floating-point data to the decimal type, it maybe overflow.
+        DataType::Float32 => Some(DataType::Decimal(14, 7)),
+        DataType::Float64 => Some(DataType::Decimal(30, 15)),
+        _ => None,
+    }
+}
+
+fn mathematics_numerical_coercion(

Review comment:
       thank you




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 removed a comment on pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 removed a comment on pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#issuecomment-1011841405


   I will rebase the master after https://github.com/apache/arrow-datafusion/pull/1483 merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 commented on pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#issuecomment-1011823462


   will rebase the master after https://github.com/apache/arrow-datafusion/pull/1483 merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] liukun4515 commented on pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
liukun4515 commented on pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#issuecomment-1011841405


   I will rebase the master after https://github.com/apache/arrow-datafusion/pull/1483 merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1554: support mathematics operation for decimal data type

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1554:
URL: https://github.com/apache/arrow-datafusion/pull/1554#discussion_r786034055



##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {
+    match numeric_type {
+        DataType::Int8 => Some(DataType::Decimal(3, 0)),
+        DataType::Int16 => Some(DataType::Decimal(5, 0)),
+        DataType::Int32 => Some(DataType::Decimal(10, 0)),
+        DataType::Int64 => Some(DataType::Decimal(20, 0)),
+        // TODO if we convert the floating-point data to the decimal type, it maybe overflow.
+        DataType::Float32 => Some(DataType::Decimal(14, 7)),
+        DataType::Float64 => Some(DataType::Decimal(30, 15)),
+        _ => None,
+    }
+}
+
+fn mathematics_numerical_coercion(

Review comment:
       This function looks almost the same as `numerical_coercion`  -- which is used for equality
   
   What is the reason for making a new function rather than extending `numerical_coercion` with Decimal support?

##########
File path: datafusion/src/physical_plan/expressions/binary.rs
##########
@@ -235,12 +236,10 @@ fn is_distinct_from_decimal(
 ) -> Result<BooleanArray> {
     let mut bool_builder = BooleanBuilder::new(left.len());
     for i in 0..left.len() {
-        if left.is_null(i) && right.is_null(i) {
-            bool_builder.append_value(false)?;
-        } else if left.is_null(i) || right.is_null(i) {
-            bool_builder.append_value(true)?;
-        } else {
-            bool_builder.append_value(left.value(i) != right.value(i))?;
+        match (left.is_null(i), right.is_null(i)) {

Review comment:
       👍 

##########
File path: datafusion/src/physical_plan/coercion_rule/binary_rule.rs
##########
@@ -162,12 +162,141 @@ fn get_comparison_common_decimal_type(
     }
 }
 
+// Convert the numeric data type to the decimal data type.
+// Now, we just support the signed integer type and floating-point type.
+fn convert_numeric_type_to_decimal(numeric_type: &DataType) -> Option<DataType> {

Review comment:
       maybe this would be better called `coerce_numeric_type_to_decimal`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org