You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "austin362667 (via GitHub)" <gi...@apache.org> on 2024/11/01 18:30:36 UTC

[PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

austin362667 opened a new pull request, #6671:
URL: https://github.com/apache/arrow-rs/pull/6671

   # Which issue does this PR close?
   
   Closes https://github.com/apache/datafusion/issues/13195
   
   # Rationale for this change
    
   Thanks to @jayzhan211 , he noticed following issue, array compute kernel `bit_length()` doesn't support `Utf8View` type:
   ```rust
   create table test_source as values
     ('Andrew', 'X'),
     ('Xiangpeng', 'Xiangpeng'),
     ('Raphael', 'R'),
     (NULL, 'R');
   create table test as
   SELECT
     arrow_cast(column1, 'Utf8View') as column1_utf8view
   FROM test_source;
   
   select bit_length(column1_utf8view) from test;
   ```
   caused the error:
   ```rust
   query error DataFusion error: Arrow error: Compute error: bit_length not supported for Utf8View
   ```
   
   # What changes are included in this PR?
   
   Update `bit_length()` array function to support `Utf8View`
   
   # Are there any user-facing changes?
   
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826861299


##########
arrow-string/src/length.rs:
##########
@@ -462,6 +487,21 @@ mod tests {
             })
     }
 
+    #[test]
+    fn bit_length_test_utf8view() {
+        bit_length_cases()

Review Comment:
   totally agree, that's what I'm thinking of, too. will add null tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1828096619


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +138,22 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = (0..list.len())
+                .map(|i| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0
+                    }
+                })
+                .collect::<Vec<u32>>();
+            Ok(Arc::new(UInt32Array::new(
+                values.into(),

Review Comment:
   Wow! Thank you so much @tustvold , you are indeed making me a better Rust programmer. Really appreciate for review. 👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826217680


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,10 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string::<i32>();

Review Comment:
   aha certainly, thanks for review!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#issuecomment-2455385346

   Got it! Thanks @tustvold I replaced `from_unary` with the [approach](https://github.com/apache/arrow-rs/pull/6671#discussion_r1827848438) you suggested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#issuecomment-2455334421

   Switching to from_unary has reverted the optimisation to not inspect the string data - https://github.com/apache/arrow-rs/pull/6671#discussion_r1827848438 is how to do this correctly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826594133


##########
arrow-string/src/length.rs:
##########
@@ -138,20 +141,12 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
         DataType::Utf8View => {
-            let string_view_array = array
-                .as_any()
-                .downcast_ref::<StringViewArray>()
-                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
-            let mut bit_lengths = Vec::with_capacity(array.len());
-            for i in 0..array.len() {
-                let bit_length = if string_view_array.is_valid(i) {
-                    (string_view_array.value(i).len() * 8) as i32
-                } else {
-                    0
-                };
-                bit_lengths.push(bit_length);
-            }
-
+            let list = array.as_string_view();
+            let bit_lengths = list
+                .views()
+                .iter()
+                .map(|view| (*view as i32) * 8)

Review Comment:
   Sure! Thanks @alamb for review



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826172416


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,10 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string::<i32>();

Review Comment:
   This will simply panic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826570445


##########
arrow-string/src/length.rs:
##########
@@ -138,20 +141,12 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
         DataType::Utf8View => {
-            let string_view_array = array
-                .as_any()
-                .downcast_ref::<StringViewArray>()
-                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
-            let mut bit_lengths = Vec::with_capacity(array.len());
-            for i in 0..array.len() {
-                let bit_length = if string_view_array.is_valid(i) {
-                    (string_view_array.value(i).len() * 8) as i32
-                } else {
-                    0
-                };
-                bit_lengths.push(bit_length);
-            }
-
+            let list = array.as_string_view();
+            let bit_lengths = list
+                .views()
+                .iter()
+                .map(|view| (*view as i32) * 8)

Review Comment:
   I also still do think you need to handle the null cases (check for which slots are null) as there might be a view of a non zero length there 



##########
arrow-string/src/length.rs:
##########
@@ -138,20 +141,12 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
         DataType::Utf8View => {
-            let string_view_array = array
-                .as_any()
-                .downcast_ref::<StringViewArray>()
-                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
-            let mut bit_lengths = Vec::with_capacity(array.len());
-            for i in 0..array.len() {
-                let bit_length = if string_view_array.is_valid(i) {
-                    (string_view_array.value(i).len() * 8) as i32
-                } else {
-                    0
-                };
-                bit_lengths.push(bit_length);
-            }
-
+            let list = array.as_string_view();
+            let bit_lengths = list
+                .views()
+                .iter()
+                .map(|view| (*view as i32) * 8)

Review Comment:
   I think the length is currently treated as a `u32` elsewhere in the code
   
   However the actual Arrow spec I think says use i32 so this is probably fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826850573


##########
arrow-string/src/length.rs:
##########
@@ -440,7 +465,7 @@ mod tests {
                 let array = StringArray::from(input);
                 let result = bit_length(&array).unwrap();
                 assert_eq!(len, result.len());
-                let result = result.as_any().downcast_ref::<Int32Array>().unwrap();
+                let result = result.as_any().downcast_ref::<UInt32Array>().unwrap();

Review Comment:
   I agree that bit_length should not beed to return signed integer, but it seems better not to change this within this PR.



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {

Review Comment:
   Why iterate over values, only to ignore them with `_`?
   use `0..(list.len())` if iterating only the indexes is desired.



##########
arrow-string/src/length.rs:
##########
@@ -115,6 +116,8 @@ pub fn length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
 /// * bit_length of null is null.
 /// * bit_length is in number of bits
 pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
+    println!("In Array bit_length()");

Review Comment:
   perhaps to remove?



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0

Review Comment:
   In https://github.com/apache/arrow-rs/pull/6662#discussion_r1826175583 it was suggested to use from_unary.
   It this applicable here?



##########
arrow-string/src/length.rs:
##########
@@ -462,6 +487,21 @@ mod tests {
             })
     }
 
+    #[test]
+    fn bit_length_test_utf8view() {
+        bit_length_cases()

Review Comment:
   We should have tests with arrays that contain null values.
   `bit_length_cases` doesn't seem to produce such.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for `bit_length` kernel [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826542917


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let string_view_array = array
+                .as_any()
+                .downcast_ref::<StringViewArray>()
+                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
+            let mut bit_lengths = Vec::with_capacity(array.len());
+            for i in 0..array.len() {
+                let bit_length = if string_view_array.is_valid(i) {
+                    (string_view_array.value(i).len() * 8) as i32

Review Comment:
   This code could be made significantly faster by just checking the lengths in the views rather than creating the string length.
   
   you can get the views like https://docs.rs/arrow/latest/arrow/array/type.StringViewArray.html#method.views
   
   The layout is described here: https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#layout-views-and-buffers
   
   Something like
   
   ```rust
   let lengths = string_view_array.views()
     .iter()
     .map(|view| view as u32)
   ```
   
   



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let string_view_array = array
+                .as_any()
+                .downcast_ref::<StringViewArray>()
+                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;

Review Comment:
   I think it would be more consistent with the rest of the codebase to use `array.as_string_view()` here rather than downcast_ref.
   
   I think this is correct, but would recommend changing the code to be consistent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#issuecomment-2455292872

   Thanks @tustvold  @alamb @findepi  for the review.
   - I roll back to `i32` rather than `u32` to make code consistent.
   - Use `from_unary` suggested from @findepi 
   - Add `bit_length_null_utf8view` test cases suggested by @tustvold .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1827848438


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +138,22 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = (0..list.len())
+                .map(|i| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0
+                    }
+                })
+                .collect::<Vec<u32>>();
+            Ok(Arc::new(UInt32Array::new(
+                values.into(),

Review Comment:
   ```suggestion
               let values = list.views().iter().map(|view| (view as i32).wrapping_mul(8)).collect();
               Ok(Arc::new(UInt32Array::new(
                   values,
   ```
   
   As we're doing infallible wrapping arithmetic, we can just compute this for everything, this will vectorize better, and the null mask is preserved below



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +138,22 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = (0..list.len())
+                .map(|i| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0
+                    }
+                })
+                .collect::<Vec<u32>>();
+            Ok(Arc::new(UInt32Array::new(
+                values.into(),

Review Comment:
   ```suggestion
               let values = list.views().iter().map(|view| (view as i32).wrapping_mul(8)).collect();
               Ok(Arc::new(Int32Array::new(
                   values,
   ```
   
   As we're doing infallible wrapping arithmetic, we can just compute this for everything, this will vectorize better, and the null mask is preserved below



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1828133873


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0

Review Comment:
   Using from_unary reverts the optimisation of only inspect the views, and not the string data.
   
   Perhaps you might use the example I wrote in https://github.com/apache/arrow-rs/pull/6671#discussion_r1827848438



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0

Review Comment:
   Using from_unary reverts the optimisation of only inspecting the views, and not the string data.
   
   Perhaps you might use the example I wrote in https://github.com/apache/arrow-rs/pull/6671#discussion_r1827848438



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1828096619


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +138,22 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = (0..list.len())
+                .map(|i| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0
+                    }
+                })
+                .collect::<Vec<u32>>();
+            Ok(Arc::new(UInt32Array::new(
+                values.into(),

Review Comment:
   Wow! Thank you so much @tustvold , you are indeed making me better Rust programmer. Really appreciate for review. 👍



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1828092955


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {
+                    if list.is_valid(i) {
+                        list.views()[i] as u32 * 8
+                    } else {
+                        0

Review Comment:
   @findepi  Thanks for bringing this up. I tried it and it works trivially well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826560668


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let string_view_array = array
+                .as_any()
+                .downcast_ref::<StringViewArray>()
+                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
+            let mut bit_lengths = Vec::with_capacity(array.len());
+            for i in 0..array.len() {
+                let bit_length = if string_view_array.is_valid(i) {
+                    (string_view_array.value(i).len() * 8) as i32

Review Comment:
   Nice! Thank you so much @alamb . Sorry.. I'm totally a newbie in StringView. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826547114


##########
arrow-string/src/length.rs:
##########
@@ -137,6 +137,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let string_view_array = array
+                .as_any()
+                .downcast_ref::<StringViewArray>()
+                .ok_or_else(|| ArrowError::ComputeError("Expected Utf8View array".to_string()))?;
+            let mut bit_lengths = Vec::with_capacity(array.len());
+            for i in 0..array.len() {
+                let bit_length = if string_view_array.is_valid(i) {
+                    (string_view_array.value(i).len() * 8) as i32

Review Comment:
   I tried to find docs on how to work with the views and I didn't find anything good - so I made a PR to add an example: https://github.com/apache/arrow-rs/pull/6673



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1826861129


##########
arrow-string/src/length.rs:
##########
@@ -115,6 +116,8 @@ pub fn length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
 /// * bit_length of null is null.
 /// * bit_length is in number of bits
 pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
+    println!("In Array bit_length()");

Review Comment:
   removed, sorry for such mistake..



##########
arrow-string/src/length.rs:
##########
@@ -137,6 +140,26 @@ pub fn bit_length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
             let list = array.as_string::<i64>();
             Ok(bit_length_impl::<Int64Type>(list.offsets(), list.nulls()))
         }
+        DataType::Utf8View => {
+            let list = array.as_string_view();
+            let values = list
+                .views()
+                .iter()
+                .enumerate()
+                .map(|(i, _)| {

Review Comment:
   of course, sorry for such mistake..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support `Utf8View` for string function `bit_length()` [arrow-rs]

Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on code in PR #6671:
URL: https://github.com/apache/arrow-rs/pull/6671#discussion_r1828093613


##########
arrow-string/src/length.rs:
##########
@@ -462,6 +487,21 @@ mod tests {
             })
     }
 
+    #[test]
+    fn bit_length_test_utf8view() {
+        bit_length_cases()

Review Comment:
   Just add the null tests, thanks @findepi 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org