You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/10/30 20:01:10 UTC

[PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

alamb opened a new pull request, #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989

   ## Which issue does this PR close?
   
   Closes https://github.com/apache/arrow-datafusion/issues/7938
   
   ## Rationale for this change
   There was a regression in query execution where multiple distinct aggregates caused a panic (because new code for `ScalarValue::new_list` was failing)
   
   ## What changes are included in this PR?
   Remove special case code in `ScalarValue::new_list` and build from existing functions
   
   ## Are these changes tested?
   New unit tests are added
   
   ## Are there any user-facing changes?
   Fix regression
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989#discussion_r1377200037


##########
datafusion/common/src/scalar.rs:
##########
@@ -1776,7 +1631,8 @@ impl ScalarValue {
             .unwrap()
     }
 
-    /// Converts `Vec<ScalaValue>` to ListArray, simplified version of ScalarValue::to_array
+    /// Converts `Vec<ScalaValue>` where each element has type corresponding to

Review Comment:
   ```suggestion
       /// Converts `Vec<ScalarValue>` where each element has type corresponding to
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989#discussion_r1376742404


##########
datafusion/common/src/scalar.rs:
##########
@@ -1802,52 +1658,12 @@ impl ScalarValue {
     /// assert_eq!(result, &expected);
     /// ```
     pub fn new_list(values: &[ScalarValue], data_type: &DataType) -> ArrayRef {
-        Arc::new(match data_type {

Review Comment:
   I am not sure why this function was implemented like this, but it is much more concisely (and completely) implemented in terms of arrays and offsets. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989#discussion_r1376745696


##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -2282,6 +2294,13 @@ select max(x_dict) from value_dict group by x_dict % 2 order by max(x_dict);
 4
 5
 
+statement ok
+drop table value

Review Comment:
   Drive By cleanup: added clean up from the test, which I noticed while working on this PR



##########
datafusion/sqllogictest/test_files/aggregate.slt:
##########
@@ -2020,14 +2020,6 @@ statement ok
 drop table t;
 
 
-
-
-statement error DataFusion error: Execution error: Table 't_source' doesn't exist\.

Review Comment:
   drive by cleanup -- this looks like it was a copy/paste of the statements immediately above, so I removed the duplication



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989#discussion_r1376786875


##########
datafusion/common/src/scalar.rs:
##########
@@ -1802,52 +1658,12 @@ impl ScalarValue {
     /// assert_eq!(result, &expected);
     /// ```
     pub fn new_list(values: &[ScalarValue], data_type: &DataType) -> ArrayRef {
-        Arc::new(match data_type {

Review Comment:
   Nice clean up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989#issuecomment-1787301589

   Thanks everyone


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix panic in multiple distinct aggregates by fixing `ScalarValue::new_list` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7989:
URL: https://github.com/apache/arrow-datafusion/pull/7989


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org