You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "izveigor (via GitHub)" <gi...@apache.org> on 2023/08/14 19:16:28 UTC
[GitHub] [arrow-datafusion] izveigor opened a new pull request, #7280: refactor: data types in `array_expressions`
izveigor opened a new pull request, #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Part of https://github.com/apache/arrow-datafusion/issues/7279
## Rationale for this change
We can use the template `decorator` (`call_array_function`) to reduce code (DRY method). I use `scoping` tactic (See examples: https://veykril.github.io/tlborm/decl-macros/minutiae/scoping.html)
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
No
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
Yes
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294963751
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -533,41 +587,12 @@ fn define_array_slice(
extra_key: &Int64Array,
return_element: bool,
) -> Result<ArrayRef> {
- match list_array.value_type() {
- DataType::List(_) => {
- slice!(list_array, key, extra_key, return_element, ListArray)
- }
- DataType::Utf8 => slice!(list_array, key, extra_key, return_element, StringArray),
- DataType::LargeUtf8 => {
- slice!(list_array, key, extra_key, return_element, LargeStringArray)
- }
- DataType::Boolean => {
- slice!(list_array, key, extra_key, return_element, BooleanArray)
- }
- DataType::Float32 => {
- slice!(list_array, key, extra_key, return_element, Float32Array)
- }
- DataType::Float64 => {
- slice!(list_array, key, extra_key, return_element, Float64Array)
- }
- DataType::Int8 => slice!(list_array, key, extra_key, return_element, Int8Array),
- DataType::Int16 => slice!(list_array, key, extra_key, return_element, Int16Array),
- DataType::Int32 => slice!(list_array, key, extra_key, return_element, Int32Array),
- DataType::Int64 => slice!(list_array, key, extra_key, return_element, Int64Array),
- DataType::UInt8 => slice!(list_array, key, extra_key, return_element, UInt8Array),
- DataType::UInt16 => {
- slice!(list_array, key, extra_key, return_element, UInt16Array)
- }
- DataType::UInt32 => {
- slice!(list_array, key, extra_key, return_element, UInt32Array)
- }
- DataType::UInt64 => {
- slice!(list_array, key, extra_key, return_element, UInt64Array)
- }
- data_type => Err(DataFusionError::NotImplemented(format!(
- "array is not implemented for types '{data_type:?}'"
- ))),
+ macro_rules! array_function {
+ ($ARRAY_TYPE:ident) => {
+ slice!(list_array, key, extra_key, return_element, $ARRAY_TYPE)
Review Comment:
this code certainly looks nicer -- thank you @izveigor
I think this function still generates an individual function for each potential `ListArray` type, when it is fundamentally an operation on indices (and thus we don't need specialized functions for Strings, Int8s, Int16s, Int32s, etc)
However this is getting closer to a single implementation so I think it is improvement to what was here previously
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294502984
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -2079,16 +1809,16 @@ macro_rules! array_has_all_non_list_check {
/// Array_has_all SQL function
pub fn array_has_all(args: &[ArrayRef]) -> Result<ArrayRef> {
- assert_eq!(args.len(), 2);
+ check_datatypes("array_has_all", &[&args[0], &args[1]])?;
- let array = args[0].as_list::<i32>();
- let sub_array = args[1].as_list::<i32>();
+ let array = as_list_array(&args[0])?;
+ let sub_array = as_list_array(&args[1])?;
let mut boolean_builder = BooleanArray::builder(array.len());
for (arr, sub_arr) in array.iter().zip(sub_array.iter()) {
if let (Some(arr), Some(sub_arr)) = (arr, sub_arr) {
- let res = match (arr.data_type(), sub_arr.data_type()) {
- (DataType::List(_), DataType::List(_)) => {
+ let res = match arr.data_type() {
Review Comment:
Is it safe to only check arr type not sub_arr type?
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -637,37 +662,20 @@ macro_rules! append {
/// Array_append SQL function
pub fn array_append(args: &[ArrayRef]) -> Result<ArrayRef> {
- if args.len() != 2 {
- return Err(DataFusionError::Internal(format!(
- "Array_append function requires two arguments, got {}",
- args.len()
- )));
- }
-
let arr = as_list_array(&args[0])?;
let element = &args[1];
- let res = match (arr.value_type(), element.data_type()) {
- (DataType::List(_), DataType::List(_)) => concat_internal(args)?,
- (DataType::Utf8, DataType::Utf8) => append!(arr, element, StringArray),
- (DataType::LargeUtf8, DataType::LargeUtf8) => append!(arr, element, LargeStringArray),
- (DataType::Boolean, DataType::Boolean) => append!(arr, element, BooleanArray),
- (DataType::Float32, DataType::Float32) => append!(arr, element, Float32Array),
- (DataType::Float64, DataType::Float64) => append!(arr, element, Float64Array),
- (DataType::Int8, DataType::Int8) => append!(arr, element, Int8Array),
- (DataType::Int16, DataType::Int16) => append!(arr, element, Int16Array),
- (DataType::Int32, DataType::Int32) => append!(arr, element, Int32Array),
- (DataType::Int64, DataType::Int64) => append!(arr, element, Int64Array),
- (DataType::UInt8, DataType::UInt8) => append!(arr, element, UInt8Array),
- (DataType::UInt16, DataType::UInt16) => append!(arr, element, UInt16Array),
- (DataType::UInt32, DataType::UInt32) => append!(arr, element, UInt32Array),
- (DataType::UInt64, DataType::UInt64) => append!(arr, element, UInt64Array),
- (DataType::Null, _) => return Ok(array(&[ColumnarValue::Array(args[1].clone())])?.into_array(1)),
- (array_data_type, element_data_type) => {
- return Err(DataFusionError::NotImplemented(format!(
- "Array_append is not implemented for types '{array_data_type:?}' and '{element_data_type:?}'."
- )))
- }
+ check_datatypes("array_append", &[arr.values(), &args[1]])?;
Review Comment:
Nit: Use `element` instead of &args[1]
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -212,6 +212,60 @@ fn compute_array_dims(arr: Option<ArrayRef>) -> Result<Option<Vec<Option<u64>>>>
}
}
+fn check_datatypes(name: &str, args: &[&ArrayRef]) -> Result<()> {
+ let data_type = args[0].data_type();
+ if !args
+ .iter()
+ .all(|arg| arg.data_type().equals_datatype(data_type))
+ {
+ return Err(DataFusionError::NotImplemented(format!(
+ "{name} is not implemented for type '{data_type:?}'."
+ )));
+ }
+
+ Ok(())
+}
+
+macro_rules! call_array_function {
+ ($DATATYPE:expr, false) => {
+ match $DATATYPE {
+ DataType::Utf8 => array_function!(StringArray),
+ DataType::LargeUtf8 => array_function!(LargeStringArray),
+ DataType::Boolean => array_function!(BooleanArray),
+ DataType::Float32 => array_function!(Float32Array),
+ DataType::Float64 => array_function!(Float64Array),
+ DataType::Int8 => array_function!(Int8Array),
+ DataType::Int16 => array_function!(Int16Array),
+ DataType::Int32 => array_function!(Int32Array),
+ DataType::Int64 => array_function!(Int64Array),
+ DataType::UInt8 => array_function!(UInt8Array),
+ DataType::UInt16 => array_function!(UInt16Array),
+ DataType::UInt32 => array_function!(UInt32Array),
+ DataType::UInt64 => array_function!(UInt64Array),
+ _ => unreachable!(),
+ }
+ };
+ ($DATATYPE:expr, $INCLUDE_LIST:expr) => {{
+ match $DATATYPE {
+ DataType::List(_) => array_function!(ListArray),
Review Comment:
Better if we use a flag to merge these two in one?
```suggestion
DataType::List(_) => if $INLCUDE_LIST {array_function!(ListArray)} else {
unreachable!()
}
```
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -1179,27 +1133,13 @@ pub fn array_positions(args: &[ArrayRef]) -> Result<ArrayRef> {
let arr = as_list_array(&args[0])?;
let element = &args[1];
- let res = match arr.value_type() {
- DataType::List(_) => positions!(arr, element, ListArray),
- DataType::Utf8 => positions!(arr, element, StringArray),
- DataType::LargeUtf8 => positions!(arr, element, LargeStringArray),
- DataType::Boolean => positions!(arr, element, BooleanArray),
- DataType::Float32 => positions!(arr, element, Float32Array),
- DataType::Float64 => positions!(arr, element, Float64Array),
- DataType::Int8 => positions!(arr, element, Int8Array),
- DataType::Int16 => positions!(arr, element, Int16Array),
- DataType::Int32 => positions!(arr, element, Int32Array),
- DataType::Int64 => positions!(arr, element, Int64Array),
- DataType::UInt8 => positions!(arr, element, UInt8Array),
- DataType::UInt16 => positions!(arr, element, UInt16Array),
- DataType::UInt32 => positions!(arr, element, UInt32Array),
- DataType::UInt64 => positions!(arr, element, UInt64Array),
- data_type => {
- return Err(DataFusionError::NotImplemented(format!(
- "Array_positions is not implemented for types '{data_type:?}'."
- )))
- }
- };
+ check_datatypes("array_positions", &[arr.values(), &args[1]])?;
Review Comment:
Nit: Use `element` instead of &args[1]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294565323
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -212,6 +212,60 @@ fn compute_array_dims(arr: Option<ArrayRef>) -> Result<Option<Vec<Option<u64>>>>
}
}
+fn check_datatypes(name: &str, args: &[&ArrayRef]) -> Result<()> {
+ let data_type = args[0].data_type();
+ if !args
+ .iter()
+ .all(|arg| arg.data_type().equals_datatype(data_type))
+ {
+ return Err(DataFusionError::NotImplemented(format!(
+ "{name} is not implemented for type '{data_type:?}'."
+ )));
+ }
+
+ Ok(())
+}
+
+macro_rules! call_array_function {
+ ($DATATYPE:expr, false) => {
+ match $DATATYPE {
+ DataType::Utf8 => array_function!(StringArray),
+ DataType::LargeUtf8 => array_function!(LargeStringArray),
+ DataType::Boolean => array_function!(BooleanArray),
+ DataType::Float32 => array_function!(Float32Array),
+ DataType::Float64 => array_function!(Float64Array),
+ DataType::Int8 => array_function!(Int8Array),
+ DataType::Int16 => array_function!(Int16Array),
+ DataType::Int32 => array_function!(Int32Array),
+ DataType::Int64 => array_function!(Int64Array),
+ DataType::UInt8 => array_function!(UInt8Array),
+ DataType::UInt16 => array_function!(UInt16Array),
+ DataType::UInt32 => array_function!(UInt32Array),
+ DataType::UInt64 => array_function!(UInt64Array),
+ _ => unreachable!(),
+ }
+ };
+ ($DATATYPE:expr, $INCLUDE_LIST:expr) => {{
+ match $DATATYPE {
+ DataType::List(_) => array_function!(ListArray),
Review Comment:
It does not work with `macro_rules!` because every function (even with `if`) will consider `List` is one of possible ways to compute function (some functions can not work with `List`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on pull request #7280: refactor: data types in `array_expressions`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#issuecomment-1680897251
@alamb I'm sorry, but we can not avoid regression with `array_append` and `array_prepend` right now. I think I can fix it in the next batch of PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#issuecomment-1680573667
🤔 the CI appears to be failing now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294965716
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -3343,6 +3036,20 @@ mod tests {
assert_eq!(result, &UInt64Array::from_value(2, 1));
}
+ #[test]
+ fn test_check_invalid_datatypes() {
+ let data = vec![Some(vec![Some(1), Some(2), Some(3)])];
+ let list_array =
+ Arc::new(ListArray::from_iter_primitive::<Int64Type, _, _>(data)) as ArrayRef;
+ let int64_array = Arc::new(StringArray::from(vec![Some("string")])) as ArrayRef;
+
+ let args = [list_array, int64_array];
+
+ let array = array_append(&args);
+
+ assert!(array.is_err());
Review Comment:
You might also want to check that the error is as expected with something like:
```suggestion
assert!(array.unwrap_err().to_string(), "Expected error message");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on pull request #7280: refactor: data types in `array_expressions`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#issuecomment-1678581696
@alamb @jayzhan211 I wonder if you have time to review the cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1295054614
##########
datafusion/core/tests/sqllogictests/test_files/array.slt:
##########
@@ -864,17 +864,18 @@ select make_array(['a','b'], null);
## array_append (aliases: `list_append`, `array_push_back`, `list_push_back`)
+# TODO: array_append with NULLS
# array_append scalar function #1
-query ?
Review Comment:
I'm now working on the problem of empty arrays, so I removed the functionality for a while. But I agree with you that it's better to leave the current null handling for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#issuecomment-1680974112
I took the liberty of merging up to this branch to resolve conflicts between this and https://github.com/apache/arrow-datafusion/pull/7293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] izveigor commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294534254
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -2079,16 +1809,16 @@ macro_rules! array_has_all_non_list_check {
/// Array_has_all SQL function
pub fn array_has_all(args: &[ArrayRef]) -> Result<ArrayRef> {
- assert_eq!(args.len(), 2);
+ check_datatypes("array_has_all", &[&args[0], &args[1]])?;
- let array = args[0].as_list::<i32>();
- let sub_array = args[1].as_list::<i32>();
+ let array = as_list_array(&args[0])?;
+ let sub_array = as_list_array(&args[1])?;
let mut boolean_builder = BooleanArray::builder(array.len());
for (arr, sub_arr) in array.iter().zip(sub_array.iter()) {
if let (Some(arr), Some(sub_arr)) = (arr, sub_arr) {
- let res = match (arr.data_type(), sub_arr.data_type()) {
- (DataType::List(_), DataType::List(_)) => {
+ let res = match arr.data_type() {
Review Comment:
Yes, `check_datatypes` checks the equality of `arr` and `sub_arr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294964457
##########
datafusion/core/tests/sqllogictests/test_files/array.slt:
##########
@@ -864,17 +864,18 @@ select make_array(['a','b'], null);
## array_append (aliases: `list_append`, `array_push_back`, `list_push_back`)
+# TODO: array_append with NULLS
# array_append scalar function #1
-query ?
Review Comment:
Is this a regression in functionality (aka something that used to work no longer does)? Is it hard to fix prior to merging this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jayzhan211 commented on a diff in pull request #7280: refactor: data types in `array_expressions`
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #7280:
URL: https://github.com/apache/arrow-datafusion/pull/7280#discussion_r1294566935
##########
datafusion/physical-expr/src/array_expressions.rs:
##########
@@ -2079,16 +1809,16 @@ macro_rules! array_has_all_non_list_check {
/// Array_has_all SQL function
pub fn array_has_all(args: &[ArrayRef]) -> Result<ArrayRef> {
- assert_eq!(args.len(), 2);
+ check_datatypes("array_has_all", &[&args[0], &args[1]])?;
- let array = args[0].as_list::<i32>();
- let sub_array = args[1].as_list::<i32>();
+ let array = as_list_array(&args[0])?;
+ let sub_array = as_list_array(&args[1])?;
let mut boolean_builder = BooleanArray::builder(array.len());
for (arr, sub_arr) in array.iter().zip(sub_array.iter()) {
if let (Some(arr), Some(sub_arr)) = (arr, sub_arr) {
- let res = match (arr.data_type(), sub_arr.data_type()) {
- (DataType::List(_), DataType::List(_)) => {
+ let res = match arr.data_type() {
Review Comment:
I see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org