You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2024/12/31 12:39:52 UTC

[PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

rluvaton opened a new pull request, #6924:
URL: https://github.com/apache/arrow-rs/pull/6924

   # Which issue does this PR close?
   
   Parts of #6923 to make it easier to implement for binary
   
   # Rationale for this change
    
   This trait helps to abstract over the different types of binary arrays
   so that we don't need to duplicate the implementation for each type.
   
   Same as `StringArrayType`
   
   # What changes are included in this PR?
   
   Added `BinaryArrayType` and implemented for `BinaryArray`, `LargeBinaryArray`, `BinaryViewArray` and `FixedSizeBinaryArray`
   
   # Are there any user-facing changes?
   
   Yes, there is a new public trait


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571388352

   > When I looked at the existing code, we have a [&'a GenericByteArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/byte_array.rs#L547) which also takes a reference and returns ArrayIter. So I wasn't sure why this was needed
   
   I think the idea is that `BinaryViewArray` is not a `GenericBinaryArray` (it is a similarly named `GenericBinaryViewArray`, right?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2583888271

   Marking as draft as I don't think this PR is waiting on feedback anymore -- it needs some test / use in the crate otherwise it risks bitrotting / maybe not working as designed.
   
   Please mark it ready for review when it is ready for another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571377122

   > Question. What does this provide which is missing with the current IntoIterator implementations for [&'a GenericByteArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/byte_array.rs#L547), [&'a FixedSizeBinaryArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/fixed_size_binary_array.rs#L651), and [&'a BinaryViewArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/byte_view_array.rs#L626)?
   
   My question above is because I think this may not be needed? But I wasn't sure if I was missing something.
   
   This introduced trait converts a referenced array to an ArrayIter, and it does so by abstracting over these three generic types (GenericBinaryArray, BinaryViewArray, FixedSizeBinaryArray) calling the `iter()`.
   
   When I looked at the existing code, we have a [&'a GenericByteArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/byte_array.rs#L547) which also takes a reference and returns ArrayIter. So I wasn't sure why this was needed -- and what it added by making the `iter()` trait which does the same thing as the existing `into_iter()` trait. Am I missing something?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571654677

   > The [above example use case](https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571397666) (kindly provided by @kylebarron) -- is already passing on main. 🤔 See demonstration: [influxdata#6](https://github.com/influxdata/arrow-rs/pull/6). I added comments for where the generic iterators are already implemented.
   
   So perhaps the conclusion is that `ArrayAccessor` is already sufficient, but it was not obvious / clear how to use it / find it.
   
   What can we do to make it clearer to find? Maybe we can add this example on `ArrayAccessor` and then add a link to `ArrayAccessor` from the documentation for `BinaryViewArray` and `BinaryArray` and `FixedSizdBinaryArray` 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571499395

   The [above example use case](https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571397666) (kindly provided by @kylebarron) -- is already passing on main. 
   See demonstration: https://github.com/influxdata/arrow-rs/pull/6.
   I added comments for where the generic iterators are already implemented.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571388104

   > When I looked at the existing code, we have a [&'a GenericByteArray](https://github.com/apache/arrow-rs/blob/b77d38d022079b106449ead3996f373edc906327/arrow-array/src/array/byte_array.rs#L547) which also takes a reference and returns ArrayIter. So I wasn't sure why this was needed -- and what it added by making the `iter()` trait which does the same thing as the existing `into_iter()` trait. Am I missing something?
   
   I am not sure -- that is why I was asking for an example of this trait being used -- then we can have a concrete discussion if this API is duplicative or not


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571397666

   IMO this looks great. It should make it easier 
   
   I was just trying to figure out what this provided that `ArrayAccessor` didn't provide directly, and I didn't realize that `ArrayAccessor` only provided `value` and `value_unchecked`. So having this `BinaryArrayType` would be nicer to be able to use iterator methods over any generic binary array input.
   
   ```rs
       fn use_binary_array<'a>(array: impl arrow_array::ArrayAccessor<Item = &'a [u8]>) {
           // Have to manually iterate over the array
           for i in 0..array.len() {
               dbg!(String::from_utf8(array.value(i).to_vec()).unwrap());
           }
       }
   
       #[test]
       fn binary_array() {
           let mut builder = BinaryViewBuilder::new();
           builder.append_value(b"foo");
           builder.append_value(b"bar");
           builder.append_value(b"baz");
           use_binary_array(&builder.finish());
   
           let mut builder = FixedSizeBinaryBuilder::new(3);
           builder.append_value(b"foo").unwrap();
           builder.append_value(b"bar").unwrap();
           builder.append_value(b"baz").unwrap();
           use_binary_array(&builder.finish());
       }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #6924:
URL: https://github.com/apache/arrow-rs/pull/6924#issuecomment-2571666110

   Well, on the contrary I was saying that `BinaryArrayType` _could be nice_ so that consumers could access iterator methods, instead of needing to iterate by hand. I'm not sure if that warrants a new public API, but given that `StringArrayType` already exists, I think having a binary corollary would make sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #6924: feat(arrow-array): add `BinaryArrayType` similar to `StringArrayType`
URL: https://github.com/apache/arrow-rs/pull/6924


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org