You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2023/08/30 14:36:48 UTC

[GitHub] [arrow] jorisvandenbossche opened a new pull request, #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

jorisvandenbossche opened a new pull request, #37467:
URL: https://github.com/apache/arrow/pull/37467

   ### Rationale for this change
   
   `MakeArrayOfNull` for list type was assuming that the values child field didn't need to be considered, but those values could also require a minimum buffer size (eg for offsets) and which could be of greater size than the list offsets if those are int32 offsets.
   
   ### Are these changes tested?
   
   Yes
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1310391926


##########
cpp/src/arrow/array/validate.cc:
##########
@@ -713,8 +713,10 @@ struct ValidateArrayImpl {
     }
 
     // An empty list array can have 0 offsets
-    const auto required_offsets = (data.length > 0) ? data.length + data.offset + 1 : 0;
     const auto offsets_byte_size = data.buffers[1]->size();
+    const auto required_offsets = ((data.length > 0) || (offsets_byte_size > 0))
+                                      ? data.length + data.offset + 1
+                                      : 0;
     if (offsets_byte_size / static_cast<int32_t>(sizeof(offset_type)) <
         required_offsets) {
       return Status::Invalid("Offsets buffer size (bytes): ", offsets_byte_size,

Review Comment:
   @pitrou do you know if this "empty list array can have 0 offsets" also applies to other types that have offsets, i.e. string types? (this was added in https://github.com/apache/arrow/pull/6230)
   
   In the current version of the PR, I am relying on ValidateFull to catch the error of creating a length-0 LargeString array with an offsets buffer of size 4, while it should be 8. 
   At the moment the offsets buffer size is not checked if the length of the array is 0, but I changed this to actually check it if the offset buffer has _some_ length (so assuming that _if_ there is an offsets buffer with a non-zero length, it should have a correct length, also if the array itself has length 0)
   
   But if it's difficult to make guarantees about this / to change ValidateFull, I can also write a dedicated test for this specific case that doesn't rely on calling ValidateFull.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1310391926


##########
cpp/src/arrow/array/validate.cc:
##########
@@ -713,8 +713,10 @@ struct ValidateArrayImpl {
     }
 
     // An empty list array can have 0 offsets
-    const auto required_offsets = (data.length > 0) ? data.length + data.offset + 1 : 0;
     const auto offsets_byte_size = data.buffers[1]->size();
+    const auto required_offsets = ((data.length > 0) || (offsets_byte_size > 0))
+                                      ? data.length + data.offset + 1
+                                      : 0;
     if (offsets_byte_size / static_cast<int32_t>(sizeof(offset_type)) <
         required_offsets) {
       return Status::Invalid("Offsets buffer size (bytes): ", offsets_byte_size,

Review Comment:
   @pitrou do you know if this "empty list array can have 0 offsets" also applies to other types that have offsets, i.e. string types?
   
   In the current version of the PR, I am relying on ValidateFull to catch the error of creating a length-0 LargeString array with an offsets buffer of size 4, while it should be 8. 
   At the moment the offsets buffer size is not checked if the length of the array is 0, but I changed this to actually check it if the offset buffer has _some_ length (so assuming that _if_ there is an offsets buffer, it should have a correct length, also if the array itself has length 0)
   
   But if it's difficult to make guarantees about this / to change ValidateFull, I can also write a dedicated test for this specific case that doesn't rely on calling ValidateFull.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #37467:
URL: https://github.com/apache/arrow/pull/37467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1341564081


##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -388,6 +388,7 @@ static std::vector<std::shared_ptr<DataType>> TestArrayUtilitiesAgainstTheseType
       large_utf8(),
       list(utf8()),
       list(int64()),  // NOTE: Regression case for ARROW-9071/MakeArrayOfNull
+      list(large_utf8()),

Review Comment:
   I added a few more nested combinations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1311583480


##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -388,6 +388,7 @@ static std::vector<std::shared_ptr<DataType>> TestArrayUtilitiesAgainstTheseType
       large_utf8(),
       list(utf8()),
       list(int64()),  // NOTE: Regression case for ARROW-9071/MakeArrayOfNull
+      list(large_utf8()),

Review Comment:
   Can we also add some nested list types into the mix?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37467:
URL: https://github.com/apache/arrow/pull/37467#issuecomment-1750279200

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 1b262a29e5cddb2534457fadfe0b77bd7767f297.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/17459883523) has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1311582571


##########
cpp/src/arrow/array/validate.cc:
##########
@@ -713,8 +713,10 @@ struct ValidateArrayImpl {
     }
 
     // An empty list array can have 0 offsets
-    const auto required_offsets = (data.length > 0) ? data.length + data.offset + 1 : 0;
     const auto offsets_byte_size = data.buffers[1]->size();
+    const auto required_offsets = ((data.length > 0) || (offsets_byte_size > 0))
+                                      ? data.length + data.offset + 1
+                                      : 0;
     if (offsets_byte_size / static_cast<int32_t>(sizeof(offset_type)) <
         required_offsets) {
       return Status::Invalid("Offsets buffer size (bytes): ", offsets_byte_size,

Review Comment:
   > @pitrou do you know if this "empty list array can have 0 offsets" also applies to other types that have offsets, i.e. string types? (this was added in #6230)
   
   I don't think so. In any case, the allowance for list types is IMO historical.
   
   > At the moment the offsets buffer size is not checked if the length of the array is 0, but I changed this to actually check it if the offset buffer has _some_ length (so assuming that _if_ there is an offsets buffer with a non-zero length, it should have a correct length, also if the array itself has length 0)
   
   That sounds like a good approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37467: GH-37437: [C++] Fix MakeArrayOfNull for list array with large string values type

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37467:
URL: https://github.com/apache/arrow/pull/37467#discussion_r1310391926


##########
cpp/src/arrow/array/validate.cc:
##########
@@ -713,8 +713,10 @@ struct ValidateArrayImpl {
     }
 
     // An empty list array can have 0 offsets
-    const auto required_offsets = (data.length > 0) ? data.length + data.offset + 1 : 0;
     const auto offsets_byte_size = data.buffers[1]->size();
+    const auto required_offsets = ((data.length > 0) || (offsets_byte_size > 0))
+                                      ? data.length + data.offset + 1
+                                      : 0;
     if (offsets_byte_size / static_cast<int32_t>(sizeof(offset_type)) <
         required_offsets) {
       return Status::Invalid("Offsets buffer size (bytes): ", offsets_byte_size,

Review Comment:
   @pitrou do you know if this "empty list array can have 0 offsets" also applies to other types that have offsets, i.e. string types?
   
   In the current version of the PR, I am relying on ValidateFull to catch the error of creating a length-0 LargeString array with an offsets buffer of size 4, while it should be 8. 
   At the moment the offsets buffer size is not checked if the length of the array is 0, but I changed this to actually check it if the offset buffer has _some_ length (so assuming that _if_ there is an offsets buffer with a non-zero legth, it should have a correct length, also if the array itself has length 0)
   
   But if it's difficult to make guarantees about this / to change ValidateFull, I can also write a dedicated test for this specific case that doesn't rely on calling ValidateFull.



##########
cpp/src/arrow/array/validate.cc:
##########
@@ -713,8 +713,10 @@ struct ValidateArrayImpl {
     }
 
     // An empty list array can have 0 offsets
-    const auto required_offsets = (data.length > 0) ? data.length + data.offset + 1 : 0;
     const auto offsets_byte_size = data.buffers[1]->size();
+    const auto required_offsets = ((data.length > 0) || (offsets_byte_size > 0))
+                                      ? data.length + data.offset + 1
+                                      : 0;
     if (offsets_byte_size / static_cast<int32_t>(sizeof(offset_type)) <
         required_offsets) {
       return Status::Invalid("Offsets buffer size (bytes): ", offsets_byte_size,

Review Comment:
   @pitrou do you know if this "empty list array can have 0 offsets" also applies to other types that have offsets, i.e. string types?
   
   In the current version of the PR, I am relying on ValidateFull to catch the error of creating a length-0 LargeString array with an offsets buffer of size 4, while it should be 8. 
   At the moment the offsets buffer size is not checked if the length of the array is 0, but I changed this to actually check it if the offset buffer has _some_ length (so assuming that _if_ there is an offsets buffer with a non-zero length, it should have a correct length, also if the array itself has length 0)
   
   But if it's difficult to make guarantees about this / to change ValidateFull, I can also write a dedicated test for this specific case that doesn't rely on calling ValidateFull.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org