You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "goldmedal (via GitHub)" <gi...@apache.org> on 2024/07/09 17:39:40 UTC
[PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
goldmedal opened a new pull request, #11361:
URL: https://github.com/apache/datafusion/pull/11361
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #11268.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674992933
##########
datafusion/functions/src/core/map.rs:
##########
@@ -235,6 +203,25 @@ impl ScalarUDFImpl for MakeMap {
}
}
+ let mut result = Vec::new();
+ for _ in 0..arg_types.len() / 2 {
+ result.push(key_type.clone());
+ result.push(value_type.clone());
Review Comment:
No, I think it works well because the following case is passed.
```
query ?
SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]);
----
{POST: 41, HEAD: 33, PATCH: }
```
Because the type of all values or all keys should be the same, after the type checking, I build a new vec based on the final key type and final value type. In this case, the original types are
```
[DataType::Utf8, DataType::Int32, DataType::Utf8,, DataType::Int32, DataType::Utf8,, DataType::Null]
```
It would be transformed to
```
[DataType::Utf8, DataType::Int32, DataType::Utf8,, DataType::Int32, DataType::Utf8,, DataType::Int32]
```
So, I think we can get `Int32(Null)` when invoking, and then create a type-correct null array for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1672721834
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
Review Comment:
these conditions should have been checked by the planner, so it would probably be ok to panic here or return an internal error. A real error is ok too, but I suspect it would be impossible to actually hit
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
+ if value_type.is_null() {
+ value_type = chunk[1].data_type();
+ }
+ else if chunk[1].data_type() != value_type {
+ return exec_err!(
+ "map requires all values to have the same type {}, got {} instead at position {}",
+ value_type,
+ chunk[1].data_type(),
+ i
+ );
+ }
+ }
+ Ok((chunk[0].clone(), chunk[1].clone()))
+ })
+ .collect::<Result<Vec<_>>>()?.into_iter().unzip();
+
+ let key = ColumnarValue::values_to_arrays(&key)?;
+ let value = ColumnarValue::values_to_arrays(&value)?;
+
+ let mut keys = Vec::new();
+ let mut values = Vec::new();
+
+ // Since a null scalar value be transformed into [NullArray] by ColumnarValue::values_to_arrays,
+ // `arrow_select::concat` will panic if we pass a [NullArray] and a non-null array to it.
+ // So we need to create a [NullArray] with the same data type as the non-null array.
+ let null_array = new_null_array(&value_type, 1);
+ for (key, value) in key.iter().zip(value.iter()) {
+ keys.push(key.as_ref());
+ if value.data_type().is_null() {
+ values.push(null_array.as_ref());
+ } else {
+ values.push(value.as_ref());
+ }
+ }
+ let key = match concat(&keys) {
+ Ok(key) => key,
+ Err(e) => return internal_err!("Error concatenating keys: {}", e),
+ };
+ let value = match concat(&values) {
+ Ok(value) => value,
+ Err(e) => return internal_err!("Error concatenating values: {}", e),
+ };
+ make_map_batch_internal(key, value)
+}
+
+fn make_map_batch(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "make_map requires exactly 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+ let key = get_first_array_ref(&args[0])?;
+ let value = get_first_array_ref(&args[1])?;
+ make_map_batch_internal(key, value)
+}
+
+fn get_first_array_ref(columnar_value: &ColumnarValue) -> Result<ArrayRef> {
+ match columnar_value {
+ ColumnarValue::Scalar(value) => match value {
+ ScalarValue::List(array) => Ok(array.value(0).clone()),
+ ScalarValue::LargeList(array) => Ok(array.value(0).clone()),
+ ScalarValue::FixedSizeList(array) => Ok(array.value(0).clone()),
+ _ => exec_err!("Expected array, got {:?}", value),
+ },
+ ColumnarValue::Array(array) => exec_err!("Expected scalar, got {:?}", array),
+ }
+}
+
+fn make_map_batch_internal(keys: ArrayRef, values: ArrayRef) -> Result<ColumnarValue> {
+ if keys.null_count() > 0 {
+ return exec_err!("map key cannot be null");
+ }
+
+ if keys.len() != values.len() {
+ return exec_err!("map requires key and value lists to have the same length");
+ }
+
+ let key_field = Arc::new(Field::new("key", keys.data_type().clone(), false));
+ let value_field = Arc::new(Field::new("value", values.data_type().clone(), true));
+ let mut entry_struct_buffer: VecDeque<(Arc<Field>, ArrayRef)> = VecDeque::new();
+ let mut entry_offsets_buffer = VecDeque::new();
+ entry_offsets_buffer.push_back(0);
+
+ entry_struct_buffer.push_back((Arc::clone(&key_field), Arc::clone(&keys)));
+ entry_struct_buffer.push_back((Arc::clone(&value_field), Arc::clone(&values)));
+ entry_offsets_buffer.push_back(keys.len() as u32);
+
+ let entry_struct: Vec<(Arc<Field>, ArrayRef)> = entry_struct_buffer.into();
+ let entry_struct = StructArray::from(entry_struct);
+
+ let map_data_type = DataType::Map(
+ Arc::new(Field::new(
+ "entries",
+ entry_struct.data_type().clone(),
+ false,
+ )),
+ false,
+ );
+
+ let entry_offsets: Vec<u32> = entry_offsets_buffer.into();
+ let entry_offsets_buffer = Buffer::from(entry_offsets.to_byte_slice());
+
+ let map_data = ArrayData::builder(map_data_type)
+ .len(entry_offsets.len() - 1)
+ .add_buffer(entry_offsets_buffer)
+ .add_child_data(entry_struct.to_data())
+ .build()?;
+
+ Ok(ColumnarValue::Array(Arc::new(MapArray::from(map_data))))
+}
+
+#[derive(Debug)]
+pub struct MakeMap {
+ signature: Signature,
+}
+
+impl Default for MakeMap {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl MakeMap {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::variadic_any(Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for MakeMap {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "make_map"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if arg_types.is_empty() {
+ return exec_err!(
+ "make_map requires at least one pair of arguments, got 0 instead"
+ );
+ }
+ if arg_types.len() % 2 != 0 {
+ return exec_err!(
+ "make_map requires an even number of arguments, got {} instead",
+ arg_types.len()
+ );
+ }
+
+ let key_type = &arg_types[0];
+ let mut value_type = &arg_types[1];
+
+ for (i, chunk) in arg_types.chunks_exact(2).enumerate() {
Review Comment:
Same comment here about type coercion
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
+ if value_type.is_null() {
+ value_type = chunk[1].data_type();
+ }
+ else if chunk[1].data_type() != value_type {
+ return exec_err!(
+ "map requires all values to have the same type {}, got {} instead at position {}",
Review Comment:
```suggestion
"make_map requires all values to have the same type {}, got {} instead at position {}",
```
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
Review Comment:
I found this code to do null checking / coercion somewhat confusing
I would have expected that the planner had done the cocersion once at plan time rather than doing it on all inputs
Perhaps you could implement `coerce_types` for the `map` function once
https://github.com/apache/datafusion/blob/585504a31fd7d9a44c97f3f19af42bace08b8cc3/datafusion/expr/src/udf.rs#L540-L542
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -100,3 +100,93 @@ physical_plan
statement ok
drop table table_with_map;
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', 30, 'OPTION', 29, 'GET', 27, 'PUT', 25, 'DELETE', 24) AS method_count;
+----
+{POST: 41, HEAD: 33, PATCH: 30, OPTION: 29, GET: 27, PUT: 25, DELETE: 24}
+
+query I
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33)['POST'];
+----
+41
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', null);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP('POST', null, 'HEAD', 33, 'PATCH', null);
+----
+{POST: , HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP(1, null, 2, 33, 3, null);
+----
+{1: , 2: 33, 3: }
+
+query ?
+SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
+----
+{[1, 2]: [a, b], [3, 4]: [b]}
+
+query error DataFusion error: Execution error: map requires all values to have the same type Int64, got Utf8 instead at position 1
+SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+
+query error DataFusion error: Execution error: make_map key cannot be null at position 2
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, null, 30);
+
+query error DataFusion error: Execution error: make_map requires all keys to have the same type Utf8, got Int64 instead at position 1
+SELECT MAKE_MAP('POST', 41, 123, 33,'PATCH', 30);
+
+query error
+SELECT MAKE_MAP()
+
+query error DataFusion error: Execution error: make_map requires an even number of arguments, got 3 instead
+SELECT MAKE_MAP('POST', 41, 'HEAD');
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, 30]);
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAP([[1,2], [3,4]], ['a', 'b']);
+----
+{[1, 2]: a, [3, 4]: b}
+
+query error
+SELECT MAP()
+
+query error DataFusion error: Execution error: map requires an even number of arguments, got 1 instead
+SELECT MAP(['POST', 'HEAD'])
+
+query error DataFusion error: Execution error: Expected list, large_list or fixed_size_list, got Null
+SELECT MAP(null, [41, 33, 30]);
+
+query error DataFusion error: Execution error: map requires key and value lists to have the same length
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33]);
+
+query error DataFusion error: Execution error: map key cannot be null
+SELECT MAP(['POST', 'HEAD', null], [41, 33, 30]);
+
+query ?
+SELECT MAP(make_array('POST', 'HEAD', 'PATCH'), make_array(41, 33, 30));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'FixedSizeList(3, Utf8)'), arrow_cast(make_array(41, 33, 30), 'FixedSizeList(3, Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), arrow_cast(make_array(41, 33, 30), 'LargeList(Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
Review Comment:
I think if possible we should also add tests for array values (not just scalars)
I wrote up some examples, like this:
```
# test that maps can be created from arrays
statement ok
create table t as values
('a', 1, ['k1', 'k2'], [10.0, 20.0]),
('b', 2, ['k3'], [30.0]),
('d', 4, ['k5', 'k6'], [50.0, 60.0]);
query error
select make_map(column1, column2) from t;
----
DataFusion error: Internal error: UDF returned a different number of rows than expected. Expected: 3, Got: 1.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
query error
select map(column3, column4) from t;
----
DataFusion error: Execution error: Expected scalar, got ListArray
[
StringArray
[
"k1",
"k2",
],
StringArray
[
"k3",
],
StringArray
[
"k5",
"k6",
],
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674028442
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -100,3 +100,93 @@ physical_plan
statement ok
drop table table_with_map;
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', 30, 'OPTION', 29, 'GET', 27, 'PUT', 25, 'DELETE', 24) AS method_count;
+----
+{POST: 41, HEAD: 33, PATCH: 30, OPTION: 29, GET: 27, PUT: 25, DELETE: 24}
+
+query I
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33)['POST'];
+----
+41
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', null);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP('POST', null, 'HEAD', 33, 'PATCH', null);
+----
+{POST: , HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP(1, null, 2, 33, 3, null);
+----
+{1: , 2: 33, 3: }
+
+query ?
+SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
+----
+{[1, 2]: [a, b], [3, 4]: [b]}
+
+query error DataFusion error: Execution error: map requires all values to have the same type Int64, got Utf8 instead at position 1
+SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+
+query error DataFusion error: Execution error: make_map key cannot be null at position 2
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, null, 30);
+
+query error DataFusion error: Execution error: make_map requires all keys to have the same type Utf8, got Int64 instead at position 1
+SELECT MAKE_MAP('POST', 41, 123, 33,'PATCH', 30);
+
+query error
+SELECT MAKE_MAP()
+
+query error DataFusion error: Execution error: make_map requires an even number of arguments, got 3 instead
+SELECT MAKE_MAP('POST', 41, 'HEAD');
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, 30]);
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAP([[1,2], [3,4]], ['a', 'b']);
+----
+{[1, 2]: a, [3, 4]: b}
+
+query error
+SELECT MAP()
+
+query error DataFusion error: Execution error: map requires an even number of arguments, got 1 instead
+SELECT MAP(['POST', 'HEAD'])
+
+query error DataFusion error: Execution error: Expected list, large_list or fixed_size_list, got Null
+SELECT MAP(null, [41, 33, 30]);
+
+query error DataFusion error: Execution error: map requires key and value lists to have the same length
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33]);
+
+query error DataFusion error: Execution error: map key cannot be null
+SELECT MAP(['POST', 'HEAD', null], [41, 33, 30]);
+
+query ?
+SELECT MAP(make_array('POST', 'HEAD', 'PATCH'), make_array(41, 33, 30));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'FixedSizeList(3, Utf8)'), arrow_cast(make_array(41, 33, 30), 'FixedSizeList(3, Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), arrow_cast(make_array(41, 33, 30), 'LargeList(Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
Review Comment:
Oops. I think I misunderstood how the `ColumnarValue` works. I expected it to be inputted per row, but it inputs all the rows at once.
```
[datafusion/functions/src/core/map.rs:59:5] &key = [
StringArray
[
"a",
"b",
"d",
],
]
[datafusion/functions/src/core/map.rs:60:5] &value = [
PrimitiveArray<Int64>
[
1,
2,
4,
],
]
```
I'll fix them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #11361:
URL: https://github.com/apache/datafusion/pull/11361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674235631
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -100,3 +100,93 @@ physical_plan
statement ok
drop table table_with_map;
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', 30, 'OPTION', 29, 'GET', 27, 'PUT', 25, 'DELETE', 24) AS method_count;
+----
+{POST: 41, HEAD: 33, PATCH: 30, OPTION: 29, GET: 27, PUT: 25, DELETE: 24}
+
+query I
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33)['POST'];
+----
+41
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', null);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP('POST', null, 'HEAD', 33, 'PATCH', null);
+----
+{POST: , HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP(1, null, 2, 33, 3, null);
+----
+{1: , 2: 33, 3: }
+
+query ?
+SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
+----
+{[1, 2]: [a, b], [3, 4]: [b]}
+
+query error DataFusion error: Execution error: map requires all values to have the same type Int64, got Utf8 instead at position 1
+SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+
+query error DataFusion error: Execution error: make_map key cannot be null at position 2
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, null, 30);
+
+query error DataFusion error: Execution error: make_map requires all keys to have the same type Utf8, got Int64 instead at position 1
+SELECT MAKE_MAP('POST', 41, 123, 33,'PATCH', 30);
+
+query error
+SELECT MAKE_MAP()
+
+query error DataFusion error: Execution error: make_map requires an even number of arguments, got 3 instead
+SELECT MAKE_MAP('POST', 41, 'HEAD');
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, 30]);
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAP([[1,2], [3,4]], ['a', 'b']);
+----
+{[1, 2]: a, [3, 4]: b}
+
+query error
+SELECT MAP()
+
+query error DataFusion error: Execution error: map requires an even number of arguments, got 1 instead
+SELECT MAP(['POST', 'HEAD'])
+
+query error DataFusion error: Execution error: Expected list, large_list or fixed_size_list, got Null
+SELECT MAP(null, [41, 33, 30]);
+
+query error DataFusion error: Execution error: map requires key and value lists to have the same length
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33]);
+
+query error DataFusion error: Execution error: map key cannot be null
+SELECT MAP(['POST', 'HEAD', null], [41, 33, 30]);
+
+query ?
+SELECT MAP(make_array('POST', 'HEAD', 'PATCH'), make_array(41, 33, 30));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'FixedSizeList(3, Utf8)'), arrow_cast(make_array(41, 33, 30), 'FixedSizeList(3, Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), arrow_cast(make_array(41, 33, 30), 'LargeList(Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
Review Comment:
I had some challenges in transforming the array values to the map array layout. If the given table and SQL are
```
statement ok
create table t as values
('a', 1, 'k1', 10),
('b', 2, 'k3', 30),
('d', 4, 'k5', 50);
query ?
SELECT make_map(column1, column2, column3, column4) FROM t;
----
{a: 1, k1: 10}
{b: 2, k3: 30}
{d: 4, k5: 50}
```
We'll get the keys like
```
[datafusion/functions/src/core/map.rs:37:5] &key = [
Array(
StringArray
[
"a",
"b",
"d",
],
),
Array(
StringArray
[
"k1",
"k3",
"k5",
],
),
]
```
I think I need to aggregate them into one array like
```
["a", "k1", "b", "k3", "d", "k5"]
```
And then, I can give the offsets when building MapArray
```
[0,2,4,6]
```
However, I have no good idea for array aggregation now. I handle the array value by throwing `not_impl_err` currently. Maybe we can solve it in the follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2219914336
For `make_map(k1, v1, k2, v2...)`, ideally we can arrange the order of args and call `map([k1, k2..], [v1, v2])`. I think we can arrange it in ExprPlanner, so we can have only one function `map` instead of two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221715698
> merge this PR as is and file tickets to iterate/improve on main (much as we did with the array / list implementations)
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221111658
> I noticed the map and struct of Arrow allow duplicate keys but I think this behavior is wrong.
I did some research
For `StructArray`s https://arrow.apache.org/docs/format/Columnar.html#struct-layout doesn't say anything about requiring the field names to be unique. It certainly is going to cause issues for many applications (including datafusion) if there is a struct with duplicated field names
For map, the spec says the keys should be unique (and application, aka DataFusion) enforced
https://github.com/apache/arrow/blob/5e451d85d7269d3fb9c7eaab06caece5718c40e5/format/Schema.fbs#L117-L145
> /// In this layout, the keys and values are each respectively contiguous. We do
> /// not constrain the key and value types, so the application is responsible
> /// for ensuring that the keys are hashable and unique. Whether the keys are sorted
> /// may be set in the metadata for this field.
So my suggestion is:
1. File a separate datafusion ticket about the fact that `named_struct` allows repeated field names
2. either file a new ticket (or Update this PR) to enforce the uniqueness of map key name for the `map`/`make_map` function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674831870
##########
datafusion/functions/src/core/map.rs:
##########
@@ -235,6 +203,25 @@ impl ScalarUDFImpl for MakeMap {
}
}
+ let mut result = Vec::new();
+ for _ in 0..arg_types.len() / 2 {
+ result.push(key_type.clone());
+ result.push(value_type.clone());
Review Comment:
It seems there is no coercion here, in this case we don't even need `coerce_types`.
Shouldn't we convert null to non-null here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2224937367
Thanks @jayzhan211 @alamb. If needed, I can help to file the follow-up issues tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2224793129
> TODOs in the next PRs
>
> * [ ] Decide if we want to have make_map follow named_struct or if it would be better to have something more like what duckdb does
> * [ ] document map/make_map in the documentation. We can do this as a follow on PR: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#struct-functions
> * [ ] Support arrays (not just scalars)
>
I think the uniqueness of map key name is a todo ticket, too.
> I think we can simplify the two map functions to one first.
As we discussed about supporting Map literal in https://github.com/apache/datafusion/issues/11268#issuecomment-2211125762, perhaps we can do it at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674058650
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
Review Comment:
It seems that you are trying to replace null with the non-null type here. It is possible to coerce_types earlier before `invoke()`
You can use `Signature::user_defined(Volatility::Immutable)` with your defined `coerce_types` to coerce null to non-null, and you can check if all the value type is the same here as well. This function is called before `return_types` and `invoke`, so we don't deal with null type after. Note that although the null type is converted to other type like Int32, but the array value is still `null`, it is something like Int32(Null).
```
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
let mut dt = DataType::Int32;
for (i, t) in arg_types.iter().enumerate() {
if i % 2 == 1 && !t.is_null() {
dt = t.clone();
}
}
let mut dts = vec![];
for (i, t) in arg_types.iter().enumerate() {
if i % 2 == 1 && t.is_null() {
dts.push(dt.clone())
} else {
dts.push(t.clone())
}
}
Ok(dts)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2225331434
> Thanks @jayzhan211 @alamb. If needed, I can help to file the follow-up issues tonight.
Thank yoU @goldmedal -- that would be awesome. I started collecting Map related tickets on an epic -- perhaps you could add the tickets you file there as well: https://github.com/apache/datafusion/issues/11429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674028442
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -100,3 +100,93 @@ physical_plan
statement ok
drop table table_with_map;
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', 30, 'OPTION', 29, 'GET', 27, 'PUT', 25, 'DELETE', 24) AS method_count;
+----
+{POST: 41, HEAD: 33, PATCH: 30, OPTION: 29, GET: 27, PUT: 25, DELETE: 24}
+
+query I
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33)['POST'];
+----
+41
+
+query ?
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, 'PATCH', null);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP('POST', null, 'HEAD', 33, 'PATCH', null);
+----
+{POST: , HEAD: 33, PATCH: }
+
+query ?
+SELECT MAKE_MAP(1, null, 2, 33, 3, null);
+----
+{1: , 2: 33, 3: }
+
+query ?
+SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
+----
+{[1, 2]: [a, b], [3, 4]: [b]}
+
+query error DataFusion error: Execution error: map requires all values to have the same type Int64, got Utf8 instead at position 1
+SELECT MAKE_MAP('POST', 41, 'HEAD', 'ab', 'PATCH', 30);
+
+query error DataFusion error: Execution error: make_map key cannot be null at position 2
+SELECT MAKE_MAP('POST', 41, 'HEAD', 33, null, 30);
+
+query error DataFusion error: Execution error: make_map requires all keys to have the same type Utf8, got Int64 instead at position 1
+SELECT MAKE_MAP('POST', 41, 123, 33,'PATCH', 30);
+
+query error
+SELECT MAKE_MAP()
+
+query error DataFusion error: Execution error: make_map requires an even number of arguments, got 3 instead
+SELECT MAKE_MAP('POST', 41, 'HEAD');
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, 30]);
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]);
+----
+{POST: 41, HEAD: 33, PATCH: }
+
+query ?
+SELECT MAP([[1,2], [3,4]], ['a', 'b']);
+----
+{[1, 2]: a, [3, 4]: b}
+
+query error
+SELECT MAP()
+
+query error DataFusion error: Execution error: map requires an even number of arguments, got 1 instead
+SELECT MAP(['POST', 'HEAD'])
+
+query error DataFusion error: Execution error: Expected list, large_list or fixed_size_list, got Null
+SELECT MAP(null, [41, 33, 30]);
+
+query error DataFusion error: Execution error: map requires key and value lists to have the same length
+SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33]);
+
+query error DataFusion error: Execution error: map key cannot be null
+SELECT MAP(['POST', 'HEAD', null], [41, 33, 30]);
+
+query ?
+SELECT MAP(make_array('POST', 'HEAD', 'PATCH'), make_array(41, 33, 30));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'FixedSizeList(3, Utf8)'), arrow_cast(make_array(41, 33, 30), 'FixedSizeList(3, Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
+
+query ?
+SELECT MAP(arrow_cast(make_array('POST', 'HEAD', 'PATCH'), 'LargeList(Utf8)'), arrow_cast(make_array(41, 33, 30), 'LargeList(Int64)'));
+----
+{POST: 41, HEAD: 33, PATCH: 30}
Review Comment:
Oops. I think I misunderstood how the `ColumnarValue` works. I expected it to be inputted per row, but it inputs all the rows at once.
```
[datafusion/functions/src/core/map.rs:59:5] &key = [
StringArray
[
"a",
"b",
"d",
],
]
[datafusion/functions/src/core/map.rs:60:5] &value = [
PrimitiveArray<Int64>
[
1,
2,
4,
],
]
```
I'll fix them. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674247820
##########
datafusion/sqllogictest/test_files/map.slt:
##########
@@ -131,19 +131,19 @@ SELECT MAKE_MAP([1,2], ['a', 'b'], [3,4], ['b']);
----
{[1, 2]: [a, b], [3, 4]: [b]}
-query error DataFusion error: Error during planning: Execution error: User-defined coercion failed with Execution\("map requires all values to have the same type Int64, got Utf8 instead at position 1"\)(.|\n)*
Review Comment:
This pattern works in my local environment but doesn't work for CI. I'm not sure of the reason currently. Just simplify the assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2224920942
I merge this first, thanks @goldmedal and @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221121316
> > For `make_map(k1, v1, k2, v2...)`, ideally we can arrange the order of args and call `map([k1, k2..], [v1, v2])`. I think we can arrange it in ExprPlanner, so we can have only one function `map` instead of two.
>
> Thanks @jayzhan211. Sounds great. Curiously, what's the benefit of moving to `ExprPlanner`? I'm not sure, but I think it just moves the cost of aggregating column values to `ExprPlanner`, right? Or is it possible to get better performance?
I agree with @jayzhan211 it seems less than ideal to have two functions rather than just one. However I agree with @goldmedal that it isn't clear that this is all that much better.
It looks to me like duckdb has several functions to create `map`s as well. https://duckdb.org/docs/sql/data_types/map
They support this kind of literal syntax
```sql
SELECT MAP {'key1': 10, 'key2': 20, 'key3': 30};
```
As well as the parallel lists implementation
```sql
SELECT MAP(['key1', 'key2', 'key3'], [10, 20, 30]);
```
They also have a function similar to `make_map` like this:
```sql
SELECT map_from_entries([('key1', 10), ('key2', 20), ('key3', 30)]);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221710127
> > Thanks @jayzhan211. Sounds great. Curiously, what's the benefit of moving to `ExprPlanner`? I'm not sure, but I think it just moves the cost of aggregating column values to `ExprPlanner`, right? Or is it possible to get better performance?
>
> I agree with @jayzhan211 it seems less than ideal to have two functions rather than just one. However I agree with @goldmedal that it isn't clear that this is all that much better.
>
> It looks to me like duckdb has several functions to create `map`s as well. https://duckdb.org/docs/sql/data_types/map
>
> They support this kind of literal syntax
>
> ```sql
> SELECT MAP {'key1': 10, 'key2': 20, 'key3': 30};
> ```
>
> As well as the parallel lists implementation
>
> ```sql
> SELECT MAP(['key1', 'key2', 'key3'], [10, 20, 30]);
> ```
>
> They also have a function similar to `make_map` like this:
>
> ```sql
> SELECT map_from_entries([('key1', 10), ('key2', 20), ('key3', 30)]);
> ```
I think we can have several functions frontend but one single function in functions crate, with ExprPlanner, we can arrange the args for the single MapFunc. Also, the biggest reason why `make_map` is slow is because the way we rearrange args involved tons of clone, it is relatively easier if we rearrange args in ExprPlanner because it is the place we collect args.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1673989968
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
Review Comment:
The main purpose of this check is to decide the type of value array to create the null array in L76. If the values are
```
[null, null, 1, 2, 3]
```
We will know the value type by the third element. So, I think the null check is necessary. However, I agree the other check isn't necessary. Maybe I don't need to implement `coerce_types` since I guess `return_type` has checked them during planning, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221830803
> So my suggestion is:
>
> 1. File a separate datafusion ticket about the fact that `named_struct` allows repeated field names
> 2. either file a new ticket (or Update this PR) to enforce the uniqueness of map key name for the `map`/`make_map` function
Thanks @alamb. I prefer to file new tickets for them.
> Some initial thoughts on things left to do:
>
> 1. document `map`/`make_map` in the documentation. We can do this as a follow on PR: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#struct-functions
> 2. Support arrays (not just scalars)
> 3. Decide if we want to have `make_map` follow `named_struct` or if it would be better to have something more like what duckdb does
>
I'll address the comments for the code today, and then file the following issues for it. (maybe after merging?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674067650
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
Review Comment:
It looks nice. Many thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1675328853
##########
datafusion/functions/src/core/map.rs:
##########
@@ -235,6 +203,25 @@ impl ScalarUDFImpl for MakeMap {
}
}
+ let mut result = Vec::new();
+ for _ in 0..arg_types.len() / 2 {
+ result.push(key_type.clone());
+ result.push(value_type.clone());
Review Comment:
Ok, I see the code above L206, it was collapsed so I miss it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2220709681
> For `make_map(k1, v1, k2, v2...)`, ideally we can arrange the order of args and call `map([k1, k2..], [v1, v2])`. I think we can arrange it in ExprPlanner, so we can have only one function `map` instead of two.
Thanks @jayzhan211. Sounds great.
Curiously, what's the benefit of moving to `ExprPlanner`? I'm not sure, but I think it just moves the cost of aggregating column values to `ExprPlanner`, right? Or is it possible to get better performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#discussion_r1674058650
##########
datafusion/functions/src/core/map.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use arrow::array::{new_null_array, Array, ArrayData, ArrayRef, MapArray, StructArray};
+use arrow::compute::concat;
+use arrow::datatypes::{DataType, Field, SchemaBuilder};
+use arrow_buffer::{Buffer, ToByteSlice};
+use datafusion_common::Result;
+use datafusion_common::{exec_err, internal_err, ScalarValue};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+
+fn make_map(args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.is_empty() {
+ return exec_err!("map requires at least one pair of arguments, got 0 instead");
+ }
+
+ if args.len() % 2 != 0 {
+ return exec_err!(
+ "map requires an even number of arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let mut value_type = args[1].data_type();
+
+ let (key, value): (Vec<_>, Vec<_>) = args.chunks_exact(2)
+ .enumerate()
+ .map(|(i, chunk)| {
+ if chunk[0].data_type().is_null() {
+ return exec_err!("map key cannot be null");
+ }
+ if !chunk[1].data_type().is_null() {
Review Comment:
It seems that you are trying to replace null with the non-null type here. It is possible to coerce_types earlier before `invoke()`
You can use `Signature::user_defined(Volatility::Immutable)` with your defined `coerce_types` to coerce null to non-null, and you can check if all the value type is the same here as well. This function is called before `return_types` and `invoke`, so we don't deal with null type after. Note that although the null type is converted to other type like Int32, but the array value is still `null`, it is something like Int32(Null).
```rust
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
let mut dt = DataType::Int32;
for (i, t) in arg_types.iter().enumerate() {
if i % 2 == 1 && !t.is_null() {
dt = t.clone();
}
}
let mut dts = vec![];
for (i, t) in arg_types.iter().enumerate() {
if i % 2 == 1 && t.is_null() {
dts.push(dt.clone())
} else {
dts.push(t.clone())
}
}
Ok(dts)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2224743438
Not sure is there any things else to do before merging this
TODOs in the next PRs
- [ ] Decide if we want to have make_map follow named_struct or if it would be better to have something more like what duckdb does
- [ ] document map/make_map in the documentation. We can do this as a follow on PR: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#struct-functions
- [ ] Support arrays (not just scalars)
I think we can simplify the two map functions to one first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2218411989
I noticed the map and struct of Arrow allow duplicate keys but I think this behavior is wrong.
```
> select named_struct('abc', 1, 'abc', 2);
+---------------------------------------------------------+
| named_struct(Utf8("abc"),Int64(1),Utf8("abc"),Int64(2)) |
+---------------------------------------------------------+
| {abc: 1, abc: 2} |
+---------------------------------------------------------+
1 row(s) fetched.
```
Typically, a struct or map should not have a duplicate key (It would cause some problems when getting data). In DuckDB, the query will fail if exist the duplicate keys.
```
D select map_from_entries([('a',null),('a',2)]);
Invalid Input Error: Map keys must be unique.
```
I'm unsure where we should handle this issue (DataFusion or Arrow). To solve it, I prefer to fix `MapArray` and `StructArray` on the Arrow side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #11361:
URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2221821217
> I think we can have several functions frontend but one single function in functions crate, with ExprPlanner, we can arrange the args for the single MapFunc. Also, the biggest reason why `make_map` is slow is because the way we rearrange args involved tons of clone, it is relatively easier if we rearrange args in ExprPlanner because it is the place we collect args.
>
> What the MapFunc's args look like depend on what is more efficient more Arrow Array. `SELECT MAP(['key1', 'key2', 'key3'], [10, 20, 30]);` seems to be the most efficient one because we don't need any other args arrangement
Thanks
It's a good reason for me. I can try it in the next PR (to support SQL syntax) and do some benchmarks for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org