You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/11/27 10:01:43 UTC
[GitHub] [arrow-rs] matthewmturner opened a new pull request #984: Add comparison kernels for DictionaryArray
matthewmturner opened a new pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984
# Which issue does this PR close?
<!---
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #869
# Rationale for this change
<!---
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!---
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!---
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992897842
> the scalar API it is not IP-cleared it was added to arrow2 post IP clearance :/
Right -- I don't understand the implications of this statement (like if code is apache licensed, that means it can be incorporated into other projects, right? So I don't really understand the IP clearance need but I remember it was complicated 🤷
On the topic of `scalars` -- there is a recent related mailing list discussion of adding "Constant Array" support to arrow which perhaps could serve the same / similar purpose: https://lists.apache.org/thread/49qzofswg1r5z7zh39pjvd1m2ggz2kdq
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992897842
> the scalar API it is not IP-cleared it was added to arrow2 post IP clearance :/
Right -- I don't understand the implications of this statement (like if code is apache licensed, that means it can be incorporated into other projects, right?). So I don't really understand the IP clearance need but I remember it was complicated 🤷
On the topic of `scalars` -- there is a recent related mailing list discussion of adding "Constant Array" support to arrow which perhaps could serve the same / similar purpose: https://lists.apache.org/thread/49qzofswg1r5z7zh39pjvd1m2ggz2kdq
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992828156
For reference (for someone not following the gist), a `DictionaryArray` is essentially
```rust
struct DictionaryArray<K> {
indices: PrimitiveArray<K>
values: Arc<dyn Array>
}
```
that is itemwise represented as `indices.into_iter().map(|optional_index| optional_index.map(|index| "values[index]"))` where `"values[index]"` represents the `index` of values.
@alamb 's insight is that we can compare a dictionary with a scalar of the same (dynamic) type as `values` by comparing `values` with that scalar (ignoring indices).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner edited a comment on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991260739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758576359
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
Makes sense, thanks for explanation. I am looking into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993938517
BTW for some context, the comparison kernels for dictionaries are very important (though not critically urgent) for our usecase in IOx. as we have large amounts of dictionary string data.
Thus given you are working on this I can reallocate some non trivial amount of my time to help. Thank you for all the effort so far @matthewmturner
cc @pauldix @jacob-marble
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989125953
> I wonder if a SIMD implementation could be done as well
If we take the approach I tried to sketch in https://github.com/apache/arrow-rs/pull/984#discussion_r765173303 i think the comparisons themselves could be vectorized (ideally by falling back to the existing vectorized kernel)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] yordan-pavlov commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
yordan-pavlov commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989120472
I wonder if a SIMD implementation could be done as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991260739
@alamb ive started updating to your proposed approach but now when testing i get an error that a type annotation is needed for type parameter T. Im still playing around with it but ran out of time for today. Does that mean that the scalar needs to be casted to an `ArrowPrimitiveType` that is then used here:
```
eq_scalar(as_primitive_array::<T>(left.values()), right)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] shepmaster commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
shepmaster commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-996007577
If it makes sense to use `From`, then there should be no material difference. You can still take a generic `T: Into<ArrowNumericType>` to avoid forcing the caller to do extra typing.
However, I don’t think that path is good as you’d still need to provide explicit types: what is the concrete type you are converting into?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-980534418
@alamb I started the work on this. Would you be able to give it a quick look when you get the chance to make sure its going in the right direction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r760595599
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
I am very sorry for confusing this conversation with mentioning `dyn_xx_lit`.
What I was (inarticulately) trying to say was that once you have `eq_dict_scalar` (and you will likely also need `eq_dict_scalar_utf8`) we will end up with several different ways to compare an array to a scalar, depending on the array type
So I was thinking ahead to adding functions like
```rust
fn eq_scalar_utf8_dyn(array: dyn &Array, right: &str) -> Boolean {
// do dispatch to the right kernel based on type of array
}
```
But definitely not for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r765173303
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -1200,6 +1257,29 @@ where
return compare_op_scalar!(left, right, |a, b| a == b);
}
+/// Perform `left == right` operation on a [`PrimitiveArray`] and a scalar value.
+pub fn eq_dict_scalar<T>(
+ left: &DictionaryArray<T>,
+ right: T::Native,
+) -> Result<BooleanArray>
+where
+ T: ArrowNumericType,
+{
+ #[cfg(not(feature = "simd"))]
+ println!("{}", std::any::type_name::<T>());
+ return compare_dict_op_scalar!(left, T, right, |a, b| a == b);
+}
Review comment:
@matthewmturner this is what I was trying to say.
I think the way you have this function with a single `T` generic parameter means one could not compare a `DictionaryArray<Int8>` (aka that has keys / indexes of `Int8`) that had values of type `DataType::Unt16`
Here is a sketch of how this might work:
```rust
/// Perform `left == right` operation on a [`DictionaryArray`] and a numeric scalar value.
pub fn eq_dict_scalar<T, K>(
left: &DictionaryArray<K>,
right: T::Native,
) -> Result<BooleanArray>
where
T: ArrowNumericType,
K: ArrowNumericType,
{
// compare to the dictionary values (e.g if the dictionary is {A,
// B} and the keys are {1,0,1,1} that represents the values B, A,
// B, B.
//
// So we compare just the dictionary {A, B} values to `right` and
//
// TODO macro-ize this
let dictionary_comparison = match left.values().data_type() {
DataType::Int8 => {
eq_scalar(as_primitive_array::<T>(left.values()), right)
}
// TODO fill in Int16, Int32, etc
_ => unimplemented!("Should error: dictionary did not store values of type T")
}?;
// Required for safety below
assert_eq!(dictionary_comparison.len(), left.values().len());
// Now, look up the dictionary for each output
let result: BooleanArray = left.keys()
.iter()
.map(|key| {
// figure out how the dictionary element at this index
// compared to the scalar
key.map(|key| {
// safety: the original array's indices were valid
// `(0 .. left.values().len()` and dictionary_comparisoon
// is the same size, checked above
unsafe {
// it would be nice to avoid checking the conversion each time
let key = key.to_usize().expect("Dictionary index not usize");
dictionary_comparison.value_unchecked(key)
}
})
})
.collect();
Ok(result)
}
```
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,54 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $T:ident, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left.values();
+
+ let array = values
+ .as_any()
+ .downcast_ref::<PrimitiveArray<$T>>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison: Vec<bool> = (0..array.len())
Review comment:
I didn't think the `values()` (the dictionary size) has to the same as the size of the overall array 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992580033
> @alamb ive started updating to your proposed approach but now when testing i get an error that a type annotation is needed for type parameter T. Im still playing around with it but ran out of time for today. Does that mean that the scalar needs to be casted to an ArrowPrimitiveType that is then used here:
@matthewmturner -- I was able to make your branch compile using this change, but that seems quite non-ideal
```diff
diff --git a/arrow/src/compute/kernels/comparison.rs b/arrow/src/compute/kernels/comparison.rs
index f98e15d549..59d960220d 100644
--- a/arrow/src/compute/kernels/comparison.rs
+++ b/arrow/src/compute/kernels/comparison.rs
@@ -2135,7 +2135,7 @@ mod tests {
builder.append_null().unwrap();
builder.append(223).unwrap();
let array = builder.finish();
- let a_eq = eq_dict_scalar(&array, 123).unwrap();
+ let a_eq = eq_dict_scalar::<UInt8Type, UInt8Type>(&array, 123).unwrap();
assert_eq!(
a_eq,
BooleanArray::from(vec![Some(true), None, Some(false)])
```
I had been hoping something like this would work
```rust
let a_eq = eq_dict_scalar(&array, 123u8).unwrap();
```
(as in ensure that the 123 was typed as a u8)
But the compiler still says it needs type annotations.
So while the direct usability of `eq_dict_scalar` might be pretty low, I think it could then be used to implement `eq_dyn_scalar` (which I know I keep going on about)
@tustvold or @carols10cents do you have any ideas how we might avoid having to add type annotations to the call of `eq_dict_scalar`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992700643
The other thing I can think of, that seems a bit of a hack, but might be ok would be to take something like `impl TryInto<i128>` which covers all current native types
```rust
/// Perform `left == right` operation on an array and a numeric scalar
/// value. Supports PrimtiveArrays, and DictionaryArrays that have primitive values
pub fn eq_dyn_scalar<T>(
left: &dyn Array,
right: T
) -> Result<BooleanArray>
where
T : TryInto<i128> + Copy + std::fmt::Debug
{
let right: i128 = right
.try_into()
.map_err(|_| ArrowError::ComputeError(format!("Can not convert scalar {:?} to i128", right)))?;
match left.data_type() {
DataType::UInt8 => {
let right: u8 = right
.try_into()
.map_err(|_| ArrowError::ComputeError(format!("Can not convert {:?} to u8 for comparison with UInt8Array", right)))?;
eq_scalar::<UInt8Type>(as_primitive::<UInt8Type>(left), right)
}
DataType::UInt16 => {
let right: u16 = right
.try_into()
.map_err(|_| ArrowError::ComputeError(format!("Can not convert {:?} to u16 for comparison with UInt16Array", right)))?;
eq_scalar::<UInt16Type>(as_primitive::<UInt16Type>(left), right)
}
// TODO macroize + fill out the other primitive array types here
DataType::Dictionary(key_type, value_type) => {
match key_type.as_ref() {
DataType::UInt8 => {
let left = as_dictionary::<UInt8Type>(left);
unpack_dict_comparison(left, eq_dyn_scalar(left.values().as_ref(), right)?)
}
// TODO fill out the rest of the key types here
_ => todo!()
}
}
_ => todo!()
}
}
/// unpacks the results of comparing left.values (as a boolean)
///
/// TODO add example
///
fn unpack_dict_comparison<K>(
left: &DictionaryArray<K>,
dict_comparison: BooleanArray,
) -> Result<BooleanArray>
where
K: ArrowNumericType,
{
assert_eq!(dict_comparison.len(), left.values().len());
let result: BooleanArray = left
.keys()
.iter()
.map(|key| {
key.map(|key| unsafe {
// safety lengths were verified above
let key = key.to_usize().expect("Dictionary index not usize");
dict_comparison.value_unchecked(key)
})
})
.collect();
Ok(result)
}
```
Which then finally allows a call to `eq_dyn_scalar` without type annotations:
```rust
#[test]
fn test_dict_eq_scalar() {
...
let array = builder.finish();
// YAY! No type annotations!
let a_eq = eq_dyn_scalar(&array, 123).unwrap();
assert_eq!(
a_eq,
BooleanArray::from(vec![Some(true), None, Some(false)])
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993936801
> @alamb ok! I can open a new issue for adding the IntoArrowNumericType trait which i think we would want to do separate from this PR. That would also give others an opportunity to provide an opinion and give a little time to see if there is path forward on the ip clearance.
Sounds good @matthewmturner .
In terms of `Scalar` / `ScalarValue` and IP clearance, here is what I would suggest:
1. We implement the kernels that dynamically dispatch on the array type `eq_dyn_scalar`, `eq_utf8_dyn_lit` and `eq_bool_dyn_lit`), -- I can definitely find time to help here)
2. We the file a follow on ticket / issue for implementing a fully dynamic kernels that dispatch both on array type and on scalar type
What can I do to be most helpful for this project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] nevi-me commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
nevi-me commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-994093090
It's been a while, but if I recall correctly, the idea was for compute kernels in arrow to follow a similar approach to C++ where we have a `Datum` (or whatever we call it), which could likely be an enum of the same datatype. Within that Datum, we'd have `Scalar`, `Array`, ...
The C++ impl has a `ChunkedArray` which is effectively `Vec<Array>`, but we never went with that.
It would have been desirable to move `Scalar` to arrow at the time, but that's fortunately been done in arrow2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992801820
> I won't claim to have read the code thoroughly, but I'm not yet understanding why there are two generics here. Could you change the function to have only one?
Sadly @shepmaster they types are different -- `K` is the type of the keys and `T` was the type of the value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995848590
@matthewmturner let me know if it would help to file a ticket for `IntoArrowNumericType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-1002581693
> Maybe worth planning for this as well.
Thanks @bkmgit -- perhaps you can file a ticket for the item ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r757885028
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
The values array can be anything (not always a `StringArray`) -- perhaps this would be a good place to use the `dyn_XX` kernels -- to compare the values array with `$right`)
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
I think one of the main points of this ticket is to avoid the call here to `vaues.value_unchecked`
I like to think about the goal in by thinking "what would happen with DictionaryArray with 1000000 entries but a dictionary of size 1?" -- the way you have this PR, I think we would call `$op` 1000000 times. The idea is to call `$op` 1 time.
So the pattern I think we are looking, at least for the constant kernels is:
In pseudo code:
```
let values = dict_array.values();
let comparison_result_on_values = apply_op_to_values();
let result = dict_array.keys().iter().map(|index| comparison_result_on_values[index]).collect()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner edited a comment on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-988092550
@alamb this took me _**much**_ longer than it should have, and im still not even sure if its idiomatic rust / arrow but i _**think**_ i might be close to getting the `eq_dict_scalar` kernel. this process did definitely help a lot with learning about rust traits / macros and how arrow uses them.
one thing i noticed is that it doesnt seem there is way to create a `DictionaryArray` from a vec of scalars like you can with a vec of str slices. Is that expected? I also dont see a builder method i can use to construct the `DictionaryArray`. I guess the main use case for `DictionaryArray` is with strings and not scalars?
Either way, would you be able to see if this update is going in the right direction? If so, then I can extend to utf8 kernels etc.
UPDATE:
Actually I found `PrimitiveDictionaryBuilder` which i think will work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992722355
IMO we need a `dyn Scalar` and a `cmp_scalar(array: &dyn Array, scalar: &dyn Scalar) -> BooleanArray` that dispatches to specific implementations based on the array's datatype. We then use `cmp_scalar(dict.values().as_ref(), scalar)` and clone the indices. Inspiration for `trait Scalar` here: https://github.com/jorgecarleitao/arrow2/tree/main/src/scalar . I hope it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] shepmaster commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
shepmaster commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992806010
Ah:
> the way you have this function with a single `T` generic parameter means one could not compare a `DictionaryArray<Int8>` (aka that has keys / indexes of `Int8`) that had values of type `DataType::Unt16`
Perhaps one possibility would be to have a trait that maps in the reverse direction? Something like
```rust
trait IntoArrowNumericType {
type Arrow: ArrowNumericType;
}
impl IntoArrowNumericType for u8 {
type Arrow = UInt8Type;
}
pub fn eq_dict_scalar<T, K>(
left: &DictionaryArray<K>,
right: T,
) -> Result<BooleanArray>
where
K: ArrowNumericType,
T: IntoArrowNumericType,
{ todo!() }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alippai commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alippai commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992996558
@alamb my understanding is that any meaningful chunk of code with any license needs to be cleared if it's not original creation of the dev written for the Apache Arrow project. I believe @jorgecarleitao referred to this: that trait is not IP cleared yet, thus it needs the legal process :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995911132
@shepmaster one quick question on your proposal.
What would be the difference between using that approach compared to implementing the `From` trait (https://doc.rust-lang.org/std/convert/trait.From.html) for `ArrowNumericType` on each of the rust native types. I guess that would change the signatures / how we call the functions to something like the following:
```
fn eq_dict_scalar<K, T>(left: &DictionaryArray<K>, right: T)
where
K: ArrowNumericType,
T: ArrowNumericType,
{
todo!()
}
// ----
fn usage<K>(d: &DictionaryArray<K>)
where
K: ArrowNumericType,
{
eq_dict_scalar(d, 8.into());
eq_dict_scalar(d, -64.into());
}
```
@alamb FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992819072
> IMO we need a dyn Scalar and a cmp_scalar(array: &dyn Array, scalar: &dyn Scalar) -> BooleanArray that dispatches to specific implementations based on the array's datatype. We then use cmp_scalar(dict.values().as_ref(), scalar) and clone the indices. Inspiration for trait Scalar here: https://github.com/jorgecarleitao/arrow2/tree/main/src/scalar . I hope it helps.
@jorgecarleitao -- Indeed, this is an excellent idea. 🤔
Following the existing pattern in the comparison kernels, we will likely end up with several kernels such as
```
eq_dyn_lit(left: &dyn Array, right: &dyn Array)
eq_utf8_dyn_lit(left: &dyn Array, right: impl AsRef<str>)
eq_bool_dyn_lit(left: &dyn Array, right: bool)
```
Which might be nice as the user's rust code could leave the scalar `right` strongly typed, but it would still have to call the correct kernel. It also has the upside that it wouldn't require an owned `String` for an Utf8 constant, but we can probably make a `dyn Scalar` work like that too with some finagling
@jorgecarleitao when you say "inspiration" do you mean it would be ok to incorporate https://github.com/jorgecarleitao/arrow2/tree/main/src/scalar into arrow-rs? I have somewhat lost track of what was needed IP clearance wise. Maybe since arrow2 is apache licensed, it is fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner edited a comment on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989103055
@alamb i made some updates and the scalar test is now passing! i think i did it right...maybe some stylistic changes to be made though. If you're okay with this implementation then i can add the other ops for scalar.
i had started on the utf8_scalar macro but im still trying to figure out if i can reuse `compare_dict_op_scalar` macro or if i need to make another one that downcasts to `GenericStringArray`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r765201351
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -1200,6 +1257,29 @@ where
return compare_op_scalar!(left, right, |a, b| a == b);
}
+/// Perform `left == right` operation on a [`PrimitiveArray`] and a scalar value.
+pub fn eq_dict_scalar<T>(
+ left: &DictionaryArray<T>,
+ right: T::Native,
+) -> Result<BooleanArray>
+where
+ T: ArrowNumericType,
+{
+ #[cfg(not(feature = "simd"))]
+ println!("{}", std::any::type_name::<T>());
+ return compare_dict_op_scalar!(left, T, right, |a, b| a == b);
+}
Review comment:
thx much for putting this together and the explanation. I'll work on implementing it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992692480
So I messed around with this some more. The key thing I kept hitting was that the comparisons to the dictionary values effectively needed to be "dynamic"
Here is one approach that we might be able to use (and skip the specialized `eq_dict_scalar` entirely).
```rust
/// Perform `left == right` operation on an array and a numeric scalar
/// value. Supports PrimtiveArrays, and DictionaryArrays that have primitive values
pub fn eq_dyn_scalar<T>(
left: &dyn Array,
right: T::Native,
) -> Result<BooleanArray>
where
T: ArrowNumericType,
{
#[cfg(not(feature = "simd"))]
println!("{}", std::any::type_name::<T>());
match left.data_type() {
DataType::UInt8 => {
// horrible (?) way to get a u8
let right: u8 = right.to_usize()
.and_then(|right| right.try_into().ok())
.ok_or_else(|| ArrowError::ComputeError(format!("Can not convert {:?} to u8 for comparison with UInt8Array", right)))?;
eq_scalar::<UInt8Type>(as_primitive::<UInt8Type>(left), right)
}
DataType::UInt16 => {
// horrible (?) way to get a u16
let right: u16 = right.to_usize()
.and_then(|right| right.try_into().ok())
.ok_or_else(|| ArrowError::ComputeError(format!("Can not convert {:?} to u16 for comparison with UInt16Array", right)))?;
eq_scalar::<UInt16Type>(as_primitive::<UInt16Type>(left), right)
}
// TODO other primitive array types
DataType::Dictionary(key_type, value_type) => {
match key_type.as_ref() {
DataType::UInt8 => {
let left = as_dictionary::<UInt8Type>(left);
unpack_dict_comparison(left, eq_dyn_scalar::<T>(left.values().as_ref(), right)?)
}
// TODO fill out the rest of the key types here
_ => todo!()
}
}
// TODO macroize / fill out rest of primitive dispatch
_ => todo!()
}
}
```
The downside is you still need type annotations at the callsite:
```rust
#[test]
fn test_dict_eq_scalar() {
...
let array = builder.finish();
// still need the UInt8Type annotations
let a_eq = eq_dyn_scalar::<UInt8Type>(&array, 123u8).unwrap();
assert_eq!(
a_eq,
BooleanArray::from(vec![Some(true), None, Some(false)])
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] yordan-pavlov commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
yordan-pavlov commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991933590
> i think the comparisons themselves could be vectorized (ideally by falling back to the existing vectorized kernel)
@alamb I thought the benefit of dictionary comparison would be that it would enable vectorization of key comparison (especially useful when values are strings), since the number of keys would usually be much larger than the number of values (although vectorization of value comparison is a nice fallback in case the dictionary isn't sorted and so a binary search wouldn't work). In your proposed approach vectorization of key comparison could still happen, but is left to the compiler (instead of using explicit vectorization from existing comparison kernels). Or have I misunderstood?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991260739
@alamb ive started updating to your proposed approach but now when testing i get an error that a type annotation is needed for type parameter T. Im still playing around with it but ran out of time for today. Does that mean that the scalar needs to be casted to an `ArrowPrimitiveType` that is then used here:
```
eq_scalar(as_primitive_array::<T>(left.values()), right)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner edited a comment on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991260739
@alamb ive started updating to your proposed approach but now when testing i get an error that a type annotation is needed for type parameter T. Im still playing around with it but ran out of time for today. Does that mean that the scalar needs to be casted to an `ArrowPrimitiveType` that is then used here:
```
eq_scalar(as_primitive_array::<T>(left.values()), right)
```
my naive view is that we are casting to a `PrimitiveArray` using an `ArrowPrimitiveType` that comes from the scalar value we want to compare. But what if the values in our `PrimitiveArray` were a larger type than the scalar (i.e. values were i64 vs scalar of i32) - shouldnt we cast the scalar to a larger type in that case?
or said differently, if we're using `eq_scalar` under the hood, which requires the `PrimitiveArray` and scalar to be of the same type, why arent we enforcing the same with `DictionaryArrays` if its ultimately being passed to a `PrimitiveArray` and using `eq_scalar`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-994078393
@alamb thank you for the additional context and all the guidance youve provided so far.
just to expand on and summarize your plan to make sure im not missing any steps, can you confirm these are the steps?
1. New issue / PR to implement `IntoArrowNumericType`
2. Use 'IntoArrowNumericType' to create the different underlying `xx_dict_scalar` functions / macros.
3. We create the dynamic array kernels (i.e. `eq_dyn_scalar`) for scalar values, using the the new dict kernels create in step 2 for `DictionaryArray` (this PR)
4. We create issue for implementing the `Scalar` api
5. We create issue for creating dynamic array and scalar kernels
Im happy to, and have enjoyed, working on this. but I dont want to slow down anything on the IOx side as a result of me getting up to speed on the different pieces here. If you're okay (and aligned on the steps above) with it we could continue as we have been on this, and ill start on the first step and ping you as I have questions / for guidance. If this isnt moving at a quick enough pace then we can work on plan for how you can assist on the implementation to speed it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993941507
Yes, what @alippai wrote. Given the number of differences that we would need to apply, re-writing it shouldn't be very difficult. My comment was really just to point out that `dyn Scalar` is a an option here. I think we discussed this some time ago with @nevi-me (the idea of a Scalar API to help us write generic code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r760303658
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
@alamb ive been reviewing this but i think i might be missing something. my understanding is that my code above is for getting the dictionary values, which can be of any type (of course above im only handling `StringArray`).
```
let values = $left
.values()
.as_any()
.downcast_ref::<StringArray>()
.unwrap()
```
But then you mention using the new `dyn_xx` kernels / creating `dyn_xx_lit` kernels. Since theres no actual compute being done here, what would the `dyn` kernels be used for? Or were you referring to using the kernels to replace more than just that section of code?
to me it looks like i need a macro to downcast `DictionaryArray.values()` into whatever type the values are, and then i could use something like `dyn_xx_lit` on that in order to get the comparison results. Is this roughly what you had in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb closed pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758736855
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
🤔 yes you are correct -- we would need to add `dyn_XX_lit` type kernels, but that seems a bit overblown
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
🤔 yes you are correct -- we would need to add `dyn_XX_lit` type kernels, but that seems a bit overkill for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995252609
Hi @matthewmturner -- I will respond tomorrow - I ran out of time today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995832987
> just to expand on and summarize your plan to make sure im not missing any steps, can you confirm these are the steps?
Yes I think those are the steps I was proposing. Thank you @matthewmturner
> If you're okay (and aligned on the steps above) with it we could continue as we have been on this, and ill start on the first step and ping you as I have questions / for guidance. If this isn't moving at a quick enough pace then we can work on plan for how you can assist on the implementation to speed it up?
Sounds like a good plan! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758757795
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
> if the primary use case for this PR was comparing dict array to constant then maybe it makes sense for me to do a separate PR for that first and then come back to this?
I think focusing on the usecase of comparing dict array to constant is the best choice for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner edited a comment on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991260739
@alamb ive started updating to your proposed approach but now when testing i get an error that a type annotation is needed for type parameter T. Im still playing around with it but ran out of time for today. Does that mean that the scalar needs to be casted to an `ArrowPrimitiveType` that is then used here:
```
eq_scalar(as_primitive_array::<T>(left.values()), right)
```
my naive view is that we are casting to a `PrimitiveArray` using an `ArrowPrimitiveType` that comes from the scalar value we want to compare. But what if the values in our `PrimitiveArray` were a larger type than the scalar (i.e. values were i64 vs scalar of i32) - shouldnt we cast the scalar to a larger type in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989103055
@alamb i made some updates the scalar test is now passing! i think i did it right...maybe some stylistic changes to be made though. If you're okay with this implementation then i can add the other ops for scalar.
i had started on the utf8_scalar macro but im still trying to figure out if i can reuse `compare_dict_op_scalar` macro or if i need to make another one that downcasts to `GenericStringArray`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993637659
@alamb ok! I can open a new issue for adding the `IntoArrowNumericType` trait which i think we would want to do separate from this PR. That would also give others an opportunity to provide an opinion and give a little time to see if there is path forward on the ip clearance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] bkmgit commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
bkmgit commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-1002151601
One area that still needs to be determined in C++ is interfaces for custom comparisons:
- https://issues.apache.org/jira/browse/ARROW-12046
- https://en.wikipedia.org/wiki/Alphabetical_order#Language-specific_conventions
- https://www.postgresql.org/docs/current/collation.html
- https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/xtfrm
Maybe worth planning for this as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-1002582502
I think we have had all the discussion on this ticket and have the follow up items tracked in other tickets / issues. Thus closing this PR down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991978376
> @alamb I thought the benefit of dictionary comparison would be that it would enable vectorization of key comparison (especially useful when values are strings), since the number of keys would usually be much larger than the number of values
Yes that is my understanding @yordan-pavlov
> In your proposed approach vectorization of key comparison could still happen, but is left to the compiler (instead of using explicit vectorization from existing comparison kernels).
I guess I was imagining that the key comparisons would be exactly as vectorized as the existing comparison kernels; Specifically, in my sketch above there is a call to `eq_scalar` (aka an existing comparison kernel)
```rust
eq_scalar(as_primitive_array::<T>(left.values()), right)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992828156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992594692
I am still thinking about this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989117706
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#984](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b5f04c5) into [master](https://codecov.io/gh/apache/arrow-rs/commit/9703f989c35b67372ee2ee7a8e70dcc9622330b0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9703f98) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #984 +/- ##
==========================================
- Coverage 82.32% 82.31% -0.01%
==========================================
Files 168 168
Lines 48717 49051 +334
==========================================
+ Hits 40107 40378 +271
- Misses 8610 8673 +63
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUva2VybmVscy9jb21wYXJpc29uLnJz) | `93.39% <100.00%> (+0.45%)` | :arrow_up: |
| [arrow/src/datatypes/native.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9uYXRpdmUucnM=) | `72.91% <0.00%> (-1.56%)` | :arrow_down: |
| [arrow/src/array/array\_string.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2FycmF5X3N0cmluZy5ycw==) | `97.08% <0.00%> (-0.83%)` | :arrow_down: |
| [parquet/src/record/reader.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvcmVjb3JkL3JlYWRlci5ycw==) | `89.83% <0.00%> (-0.63%)` | :arrow_down: |
| [parquet/src/schema/printer.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvc2NoZW1hL3ByaW50ZXIucnM=) | `72.47% <0.00%> (-0.55%)` | :arrow_down: |
| [arrow/src/util/display.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL3V0aWwvZGlzcGxheS5ycw==) | `20.00% <0.00%> (-0.39%)` | :arrow_down: |
| [arrow/src/array/transform/mod.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L3RyYW5zZm9ybS9tb2QucnM=) | `85.10% <0.00%> (-0.38%)` | :arrow_down: |
| [arrow/src/datatypes/schema.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9zY2hlbWEucnM=) | `66.66% <0.00%> (-0.27%)` | :arrow_down: |
| [arrow/src/array/array.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2FycmF5LnJz) | `85.45% <0.00%> (-0.26%)` | :arrow_down: |
| [parquet\_derive/src/parquet\_field.rs](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldF9kZXJpdmUvc3JjL3BhcnF1ZXRfZmllbGQucnM=) | `65.98% <0.00%> (-0.23%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/apache/arrow-rs/pull/984/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9703f98...b5f04c5](https://codecov.io/gh/apache/arrow-rs/pull/984?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993476432
> but would adding a dyn Scalar trait mean having to update everywhere rust native scalars are currently used with that trait?
One approach could be to remove all the rust native scalars and replace with `dyn Scalar`, but another approach is to have both (keep the existing kernels with different signatures, and add a new entirely dynamic kernel `eq_scalar(&dyn Array, &dyn Scalar`).
I think keeping with the existing style of separate kernels for separate rust native types for this PR is probable,
> of course this is assuming legal issues are resolved. on that point, independent from the context of this PR, is IP clearance expected to refresh anytime soon?
I defer to @jorgecarleitao on his plans, as he is the primary author of most of arrow2
> The approach from @shepmaster seemed quite nice as well however would still require users setting the type of the scalar - is that acceptable?
I think it is acceptable (and a good idea) personally. It would result in perhaps something like:
```rust
/// Perform `left == right` operation on an array and a numeric scalar
/// value. Supports PrimtiveArrays, and DictionaryArrays that have primitive numeric values
pub fn eq_dyn_scalar<T>(
left: &dyn Array,
right: T
) -> Result<BooleanArray>
where
T : IntoArrowNumericType
{
...
}
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-996064920
> However, I don’t think that path is good as you’d still need to provide explicit types: what is the concrete type you are converting into?
>
Actually, i think im mistaken and mixing up types and traits. I dont think we need to do a conversion. Sry about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995847925
> It's been a while, but if I recall correctly, the idea was for compute kernels in arrow to follow a similar approach to C++ where we have a Datum (or whatever we call it), which could likely be an enum of the same datatype. Within that Datum, we'd have Scalar, Array, ...
That is a great idea @nevi-me -- I couldn't find an existing ticket so I wrote a new one here: https://github.com/apache/arrow-rs/issues/1047
Maybe we can work towards that vision, starting with the equality kernels (we'll need all the pieces and tests, even when we have a fully dynamic dispatch, I suspect)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758759510
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
Ok! Will start with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r761431099
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
🤔
Yes this is definitely tricky. Maybe taking a step back, and think about the usecase: comparing DictionaryArrays to literals.
For example, if you look at the comparison kernels (for `eq`) , https://docs.rs/arrow/6.3.0/arrow/compute/kernels/comparison/index.html we find;
```
eq_scalar
eq_bool_scalar
eq_utf8_scalar
```
With each being typed based on the type of scalar (because the arrays are typed)
The issue with a `DictionaryArray` is that it could have numbers, bool, strings, etc. so we can't have a single entrypoint as we do with other types of arrays
So i am thinking we would need something like
```
eq_dict_scalar // numeric
eq_dict_bool_scalar // boolean
eq_dict_utf8_scalar // strings
```
where each of those kernels would be able to downcast the array appropriately.
However, having three functions for each dict kernel seems somewhat crazy.
That is where my dyn idea was coming from. If we are going to add three new kernels for each operator (`eq`, `lt`, etc) we could perhaps add
```
eq_dyn_scalar // numeric
eq_dyn_bool_scalar // boolean
eq_dyn_utf8_scalar // strings
``
etc
Which handle DictionaryArray as well as dispatching to the other `eq_scalar`, `eq_bool_scalar`, `eq_utf8_scalar` as appropriate.
Does that make sense? I can try and sketch out the interface this weekend sometime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r761431099
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
🤔
Yes this is definitely tricky. Maybe taking a step back, and think about the usecase: comparing DictionaryArrays to literals.
For example, if you look at the comparison kernels (for `eq`) , https://docs.rs/arrow/6.3.0/arrow/compute/kernels/comparison/index.html we find;
```
eq_scalar
eq_bool_scalar
eq_utf8_scalar
```
With each being typed based on the type of scalar (because the arrays are typed)
The issue with a `DictionaryArray` is that it could have numbers, bool, strings, etc. so we can't have a single entrypoint as we do with other types of arrays
So i am thinking we would need something like
```
eq_dict_scalar // numeric
eq_dict_bool_scalar // boolean
eq_dict_utf8_scalar // strings
```
where each of those kernels would be able to downcast the array appropriately.
However, having three functions for each dict kernel seems somewhat crazy.
That is where my dyn idea was coming from. If we are going to add three new kernels for each operator (`eq`, `lt`, etc) we could perhaps add
```
eq_dyn_scalar // numeric
eq_dyn_bool_scalar // boolean
eq_dyn_utf8_scalar // strings
```
etc
Which handle DictionaryArray as well as dispatching to the other `eq_scalar`, `eq_bool_scalar`, `eq_utf8_scalar` as appropriate.
Does that make sense? I can try and sketch out the interface this weekend sometime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-988092550
@alamb this took me _**much**_ longer than it should have, and im still not even sure if its idiomatic rust / arrow but i _**think**_ i might be close to getting the `eq_dict_scalar` kernel. this process did definitely help a lot with learning about rust traits / macros and how arrow uses them.
one thing i noticed is that it doesnt seem there is way to create a `DictionaryArray` from a vec of scalars like you can with a vec of str slices. Is that expected? I also dont see a builder method i can use to construct the `DictionaryArray`. I guess the main use case for `DictionaryArray` is with strings and not scalars?
Either way, would you be able to see if this update is going in the right direction? If so, then I can extend to utf8 kernels etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989085691
> one thing i noticed is that it doesnt seem there is way to create a DictionaryArray from a vec of scalars like you can with a vec of str slices.
You can create `DictionaryArrays` from `&str` using `FromIter` as shown in https://docs.rs/arrow/6.3.0/arrow/array/struct.DictionaryArray.html
```
use arrow::array::{DictionaryArray, Int8Array};
use arrow::datatypes::Int8Type;
let test = vec!["a", "a", "b", "c"];
let array : DictionaryArray<Int8Type> = test.iter().map(|&x| if x == "b" {None} else {Some(x)}).collect();
assert_eq!(array.keys(), &Int8Array::from(vec![Some(0), Some(0), None, Some(1)]));
```
There isn't an equivalent syntax for other scalar types (e.g. `u32`, `u64`, etc) though we could add them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] shepmaster edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
shepmaster edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992806010
Ah:
> the way you have this function with a single `T` generic parameter means one could not compare a `DictionaryArray<Int8>` (aka that has keys / indexes of `Int8`) that had values of type `DataType::Unt16`
Perhaps one possibility would be to have a trait that maps in the reverse direction? Something like
```rust
trait IntoArrowNumericType {
type Arrow: ArrowNumericType;
}
impl IntoArrowNumericType for u8 {
type Arrow = UInt8Type;
}
pub fn eq_dict_scalar<K, T>(
left: &DictionaryArray<K>,
right: T,
) -> Result<BooleanArray>
where
K: ArrowNumericType,
T: IntoArrowNumericType,
{ todo!() }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992821568
> Perhaps one possibility would be to have a trait that maps in the reverse direction? Something like playground
@shepmaster this is a very cool idea -- in fact I think that is much better than taking an `impl Into<i128>` as I suggested in https://github.com/apache/arrow-rs/pull/984#issuecomment-992700643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993196776
@alamb @jorgecarleitao im not familiar enough with the rest of the code base, but would adding a `dyn Scalar` trait mean having to update everywhere rust native scalars are currently used with that trait? of course this is assuming legal issues are resolved. on that point, independent from the context of this PR, is IP clearance expected to refresh anytime soon?
The approach from @shepmaster seemed quite nice as well however would still require users setting the type of the scalar - is that acceptable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-993938517
BTW for some context, the comparison kernels for dictionaries are very important (though not critically urgent) for our usecase in IOx. as we have large amounts of dictionary string data.
Thus given you are working on this I can reallocate some non trivial amount of my time to help. Thank you for all the effort so far @matthewmturner
cc @pauldix @jacobmarble
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-995852572
> @matthewmturner let me know if it would help to file a ticket for `IntoArrowNumericType`
@alamb Sure that would be great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r765200445
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,54 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $T:ident, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left.values();
+
+ let array = values
+ .as_any()
+ .downcast_ref::<PrimitiveArray<$T>>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison: Vec<bool> = (0..array.len())
Review comment:
if youre referring to the safety comment i just hadnt removed that yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758597314
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
from my understanding of the dyn kernels those cant be used when comparing to constant right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r758741433
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
Review comment:
if the primary use case for this PR was comparing dict array to constant then maybe it makes sense for me to do a separate PR for that first and then come back to this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-998064903
@matthewmturner - I filed https://github.com/apache/arrow-rs/issues/1068 to track the `IntoArrayNumericType` trait
Before we spend time creating polished PRs for `IntoArrowNumericType` I think we should try and spike out a PR showing how the API would work with `eq_dyn_scalar`
Once we had that basic framework in place then we could fill out the implementation in individual PRs.
If it would help, I could try and take a crack at creating such a PR (likely based on this one)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r761341465
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
@alamb im struggling with the second step in your pseudocode given that my understanding is that the values could be of any `ArrowPrimativeType`. Would you be able to provide guidance on how to handle that? I've been playing with different macros and iteration options on the underlying buffers, but i feel like im missing some fundamental understanding about how to work with dynamic data type like this or how to use `ArrayData`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on a change in pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on a change in pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#discussion_r762027157
##########
File path: arrow/src/compute/kernels/comparison.rs
##########
@@ -200,6 +201,42 @@ macro_rules! compare_op_scalar_primitive {
}};
}
+macro_rules! compare_dict_op_scalar {
+ ($left:expr, $right:expr, $op:expr) => {{
+ let null_bit_buffer = $left
+ .data()
+ .null_buffer()
+ .map(|b| b.bit_slice($left.offset(), $left.len()));
+
+ let values = $left
+ .values()
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ // Safety:
+ // `i < $left.len()`
+ let comparison = (0..$left.len()).map(|i| unsafe {
+ let key = $left.keys().value_unchecked(i).to_usize().unwrap();
+ $op(values.value_unchecked(key), $right)
+ });
Review comment:
thanks for explanation! yes, it does make sense. i think i was trying to do too much in my macros / functions which was causing my confusion. i think if i can get one of the below to work that should give me my baseline to do the rest.
```
eq_dict_scalar // numeric
eq_dict_bool_scalar // boolean
eq_dict_utf8_scalar // strings
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-989100621
The type system is very tricky -- let me see if I can sketch something out to help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992828156
For reference (for anyone not following the gist), a `DictionaryArray` is essentially
```rust
struct DictionaryArray<K> {
indices: PrimitiveArray<K>
values: Arc<dyn Array>
}
```
that is itemwise represented as `indices.into_iter().map(|optional_index| optional_index.map(|index| "values[index]"))` where `"values[index]"` represents the `index` of values.
@alamb 's insight is that we can compare a dictionary with a scalar of the same (dynamic) type as `values` by comparing `values` with that scalar (ignoring indices).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992836383
"inspiration" because
* there is Datafusion's scalar API based on enums, which we could consider instead
* in arrow2 there is a smaller number of physical types than arrow (one per physical type as opposed to one per logical type), so it can't be copied _as is_
* the scalar API it is not IP-cleared it was added to arrow2 post IP clearance :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #984: Add comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-991978461
@matthewmturner I plan to answer your question in more detail tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] shepmaster commented on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
shepmaster commented on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992777187
> avoid having to add type annotations to the call of `eq_dict_scalar`?
You can likely infer the first type (`eq_dict_scalar::<_, UInt8Type>(&array, 123)`) because the type is directly related to the `left` argument:
```rust
pub fn eq_dict_scalar<T, K>(
left: &DictionaryArray<K>,
right: T::Native,
) -> Result<BooleanArray>
where
T: ArrowNumericType,
K: ArrowNumericType,
```
The problem with the `right` argument is that there's potentially an infinite number of concrete `T` types that could have a matching associated type `Native`:
```rust
impl ArrowNumericType for OneThing {
type Native = u8;
}
impl ArrowNumericType for SomeOtherThing {
type Native = u8;
}
```
I won't claim to have read the code thoroughly, but I'm not yet understanding why there are two generics here. Could you change the function to have only one?
```rust
pub fn eq_dict_scalar<K>(
left: &DictionaryArray<K>,
right: K::Native,
) -> Result<BooleanArray>
where
K: ArrowNumericType,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] shepmaster edited a comment on pull request #984: Add scalar comparison kernels for DictionaryArray
Posted by GitBox <gi...@apache.org>.
shepmaster edited a comment on pull request #984:
URL: https://github.com/apache/arrow-rs/pull/984#issuecomment-992806010
Ah:
> the way you have this function with a single `T` generic parameter means one could not compare a `DictionaryArray<Int8>` (aka that has keys / indexes of `Int8`) that had values of type `DataType::Unt16`
Perhaps one possibility would be to have a trait that maps in the reverse direction? Something like [playground](https://play.integer32.com/?version=stable&mode=debug&edition=2021&gist=448c2ea444c8752b75e3d6c89a9d1ffb)
```rust
struct UInt8Type;
struct Int64Type;
struct DictionaryArray<K>(K);
// ----
trait ArrowNumericType {
type Native;
}
trait IntoArrowNumericType {
type Arrow: ArrowNumericType<Native = Self>;
}
// ----
impl ArrowNumericType for UInt8Type {
type Native = u8;
}
impl IntoArrowNumericType for u8 {
type Arrow = UInt8Type;
}
impl ArrowNumericType for Int64Type {
type Native = i64;
}
impl IntoArrowNumericType for i64 {
type Arrow = Int64Type;
}
// ----
fn eq_dict_scalar<K, T>(left: &DictionaryArray<K>, right: T)
where
K: ArrowNumericType,
T: IntoArrowNumericType,
{
todo!()
}
// ----
fn usage<K>(d: &DictionaryArray<K>)
where
K: ArrowNumericType,
{
eq_dict_scalar(d, 8u8);
eq_dict_scalar(d, -64i64);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org