You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "felipecrv (via GitHub)" <gi...@apache.org> on 2023/08/15 14:23:22 UTC

[GitHub] [arrow] felipecrv opened a new pull request, #37174: GH-37173: [C++] C-export/import Run-End Encoded Arrays

felipecrv opened a new pull request, #37174:
URL: https://github.com/apache/arrow/pull/37174

   ### Rationale for this change
   
   All array types should be C-exportable.
   
   ### What changes are included in this PR?
   
   Ability to C-import/expor REE arrays.
   
   ### Are these changes tested?
   
   Yes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297209254


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -31,6 +31,7 @@
 #include "arrow/c/bridge.h"
 #include "arrow/c/helpers.h"
 #include "arrow/c/util_internal.h"
+#include "arrow/compute/api_vector.h"

Review Comment:
   I will add the ifdefs and mention the issue in the code https://github.com/apache/arrow/issues/37221



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] bkietz commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1300503396


##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1601,6 +1621,23 @@ struct ArrayImporter {
     return Status::OK();
   }
 
+  Status Visit(const RunEndEncodedType& type) {
+    RETURN_NOT_OK(CheckNumChildren(2));
+    if (c_struct_->n_buffers == 1) {
+      // REE arrays are not required to have any buffer,
+      // but an empty validity bitmap buffer is allowed.
+      RETURN_NOT_OK(AllocateArrayData());

Review Comment:
   Agreed, I wouldn't expect any exporters of REE arrays to emplace the placeholder null buffer. Let's remove the above check `if (c_struct_->n_buffers == 1) {` and require zero buffers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1296192914


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -31,6 +31,7 @@
 #include "arrow/c/bridge.h"
 #include "arrow/c/helpers.h"
 #include "arrow/c/util_internal.h"
+#include "arrow/compute/api_vector.h"

Review Comment:
   >  Can I create an issue about this to be fixed outside the context of this PR? 
   
   That sounds ok to me.
   
   >  I could make RunEndEncode usable without the entirety of api_vector.h --- as a C++ API utility function that is shared between the compute kernel and internal code like the test utilities?
   
   Yep!
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1300811434


##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1601,6 +1621,23 @@ struct ArrayImporter {
     return Status::OK();
   }
 
+  Status Visit(const RunEndEncodedType& type) {
+    RETURN_NOT_OK(CheckNumChildren(2));
+    if (c_struct_->n_buffers == 1) {
+      // REE arrays are not required to have any buffer,
+      // but an empty validity bitmap buffer is allowed.
+      RETURN_NOT_OK(AllocateArrayData());

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] bkietz commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1295916233


##########
docs/source/format/CDataInterface.rst:
##########
@@ -217,6 +217,8 @@ names and types of child fields are read from the child arrays.
 +------------------------+---------------------------------------------------+------------+
 | ``+us:I,J,...``        | sparse union with type ids I,J...                 |            |
 +------------------------+---------------------------------------------------+------------+
+| ``+r``                 | run-end encoded                                   |            |

Review Comment:
   I think reiteration is warranted in this case, something like:
   ```suggestion
   | ``+r``                 | run-end encoded                                   | \(3)       |
   ```
   ```rst
   (3)
      As specified in the Arrow columnar format, the run end encoded type two children where the first is the (integral) run ends and the second is the values.
   ```



##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   Just to prove it's not a problem, I'd like to see a test or two where the values are non-primitive (IE nested and/or dictionary encoded)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on pull request #37174: GH-37173: [C++] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1679104904

   > @felipecrv Should I make the equivalent Go side changes and push them to this PR/branch? Or should I make a separate PR? Thoughts?
   
   If you need my changes to test the Go implementation, you should wait a bit cause I'm still work on this one. I should even mark this as a draft.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1679246020

   > Unfortunately at the moment there isn't yet thorough c-data integration testing (but we want to add some). so I don't need your changes to add any testing, though I'd still want to do so manually at least. Do you have a preference for me keeping the Go changes separate from this PR or contributing it to this PR?
   
   Feel free to push to this PR. I think I'm done now with the changes after fixing the buffer count issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297410899


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   Of course. The default factories in these tests made me forget about that. Writing tests with REEs of lists of strings now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1296188461


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -31,6 +31,7 @@
 #include "arrow/c/bridge.h"
 #include "arrow/c/helpers.h"
 #include "arrow/c/util_internal.h"
+#include "arrow/compute/api_vector.h"

Review Comment:
   `RunEndEncode` and decode are considered one of the basic kernels that don't need the full arrow-compute build to be available as they are so integral to doing anything with REEs. 
   
   > ...would take separate strings for run ends and values.
   
   That makes writing these tests so much harder and the code less readable. Can I create an issue about this to be fixed outside the context of this PR? I could make `RunEndEncode` usable without the entirety of `api_vector.h` --- as a C++ API utility function that is shared between the compute kernel and internal code like the test utilities?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1299804990


##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1601,6 +1621,23 @@ struct ArrayImporter {
     return Status::OK();
   }
 
+  Status Visit(const RunEndEncodedType& type) {
+    RETURN_NOT_OK(CheckNumChildren(2));
+    if (c_struct_->n_buffers == 1) {
+      // REE arrays are not required to have any buffer,
+      // but an empty validity bitmap buffer is allowed.
+      RETURN_NOT_OK(AllocateArrayData());

Review Comment:
   I initially missed this, but is there a reason we want to allow this? We don't have any legacy implementations to worry about for this new datatype.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1689976278

   @zeroshade Would you like to add REE to the roundtripped types in `test_export_to_cgo.py`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1303214836


##########
go/arrow/cdata/cdata_test_framework.go:
##########
@@ -277,6 +283,12 @@ func createCArr(arr arrow.Array) *CArrowArray {
 	carr.length = C.int64_t(arr.Len())
 	carr.null_count = C.int64_t(arr.NullN())
 	carr.offset = C.int64_t(arr.Data().Offset())
+	carr.release = (*[0]byte)(C.release_test_arr)
+
+	if !arrow.IsUnion(arr.DataType().ID()) && !internal.DefaultHasValidityBitmap(arr.DataType().ID()) {

Review Comment:
   I meant more something along the lines of `len(buffers) > 0`. Why do we have to special-case this for unions and the validity bitmap? Or is this `if` clause even useful?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on a diff in pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1303226376


##########
go/arrow/cdata/cdata_test_framework.go:
##########
@@ -277,6 +283,12 @@ func createCArr(arr arrow.Array) *CArrowArray {
 	carr.length = C.int64_t(arr.Len())
 	carr.null_count = C.int64_t(arr.NullN())
 	carr.offset = C.int64_t(arr.Data().Offset())
+	carr.release = (*[0]byte)(C.release_test_arr)
+
+	if !arrow.IsUnion(arr.DataType().ID()) && !internal.DefaultHasValidityBitmap(arr.DataType().ID()) {

Review Comment:
   Let me double check. I think there are still areas of the code which assume there is at least one element in the list of buffers (even if it's nil). For REE arrays (along with others) we initialize the buffer list with a single nil. I'll look through the code and see if it's safe enough to just let the slice of buffers be empty.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297327229


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   > The best I can do is a string array because `run_end_encode` doesn't produce REE-encoded arrays of nested values.
   
   You can simply build the REE array from its components (see my suggestion for a `REEArrayFromJSON`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1684152920

   I believe I fixed all feedback points now. Please re-review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1298639458


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   I added checks on the array roundtrip to forbid nested REEs and REEs with non-supported run-end types



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #37174: GH-37173: [C++] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1679160275

   Unfortunately at the moment there isn't yet thorough c-data integration testing (but we want to add some). so I don't need your changes to add any testing, though I'd still want to do so manually at least. Do you have a preference for me keeping the Go changes separate from this PR or contributing it to this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #37174: GH-37173: [C++] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1679083660

   @felipecrv Should I make the equivalent Go side changes and push them to this PR/branch? Or should I make a separate PR? Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1296189667


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -1872,6 +1942,13 @@ TEST_F(TestSchemaImport, Map) {
   CheckImport(expected);
 }
 
+TEST_F(TestSchemaImport, RunEndEncoded) {
+  FillPrimitive(AddChild(), "s", "run_ends");
+  FillPrimitive(AddChild(), "I", "values");
+  FillRunEndEncoded("+r");
+  CheckImport(run_end_encoded(int16(), uint32()));

Review Comment:
   I will add a check and a test for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1303023093


##########
go/arrow/cdata/cdata_test_framework.go:
##########
@@ -277,6 +282,12 @@ func createCArr(arr arrow.Array) *CArrowArray {
 	carr.length = C.int64_t(arr.Len())
 	carr.null_count = C.int64_t(arr.NullN())
 	carr.offset = C.int64_t(arr.Data().Offset())
+	carr.release = (*[0]byte)(C.release_test_arr)
+
+	if arr.DataType().ID() == arrow.RUN_END_ENCODED {

Review Comment:
   Instead of testing the type id, why not test the number of buffers?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1296026622


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -31,6 +31,7 @@
 #include "arrow/c/bridge.h"
 #include "arrow/c/helpers.h"
 #include "arrow/c/util_internal.h"
+#include "arrow/compute/api_vector.h"

Review Comment:
   I don't think we depend on ARROW_COMPUTE currently for these tests. We should at least guard against it (use a `#ifdef ARROW_COMPUTE` and skip the test otherwise).



##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -1872,6 +1942,13 @@ TEST_F(TestSchemaImport, Map) {
   CheckImport(expected);
 }
 
+TEST_F(TestSchemaImport, RunEndEncoded) {
+  FillPrimitive(AddChild(), "s", "run_ends");
+  FillPrimitive(AddChild(), "I", "values");
+  FillRunEndEncoded("+r");
+  CheckImport(run_end_encoded(int16(), uint32()));

Review Comment:
   What happens if the first child is not one of the allowed run ends types? Can you add a test for it?
   (see e.g. the `DictionaryError` test below)



##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -31,6 +31,7 @@
 #include "arrow/c/bridge.h"
 #include "arrow/c/helpers.h"
 #include "arrow/c/util_internal.h"
+#include "arrow/compute/api_vector.h"

Review Comment:
   Or instead we could have a `REEArrayFromJSON` helper in https://github.com/apache/arrow/blob/main/cpp/src/arrow/ipc/json_simple.h that, just like `DictArrayFromJSON`, would take separate strings for run ends and values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1298429413


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -1872,6 +1942,13 @@ TEST_F(TestSchemaImport, Map) {
   CheckImport(expected);
 }
 
+TEST_F(TestSchemaImport, RunEndEncoded) {
+  FillPrimitive(AddChild(), "s", "run_ends");
+  FillPrimitive(AddChild(), "I", "values");
+  FillRunEndEncoded("+r");
+  CheckImport(run_end_encoded(int16(), uint32()));

Review Comment:
   I figured it out now. :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297516597


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -1872,6 +1942,13 @@ TEST_F(TestSchemaImport, Map) {
   CheckImport(expected);
 }
 
+TEST_F(TestSchemaImport, RunEndEncoded) {
+  FillPrimitive(AddChild(), "s", "run_ends");
+  FillPrimitive(AddChild(), "I", "values");
+  FillRunEndEncoded("+r");
+  CheckImport(run_end_encoded(int16(), uint32()));

Review Comment:
   I'm having a hard time producing a nested REE array with this API. @pitrou, what order of calls should I use to produce a `run_end_encoded(int32, run_end_encoded(int32, utf8))`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1300231022


##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1601,6 +1621,23 @@ struct ArrayImporter {
     return Status::OK();
   }
 
+  Status Visit(const RunEndEncodedType& type) {
+    RETURN_NOT_OK(CheckNumChildren(2));
+    if (c_struct_->n_buffers == 1) {
+      // REE arrays are not required to have any buffer,
+      // but an empty validity bitmap buffer is allowed.
+      RETURN_NOT_OK(AllocateArrayData());

Review Comment:
   The current implementation itself only works by always having the first buffer set to null because so much code assumes `buffers[0]` to be checkable. @bkietz even pinged me about some issues he had when this wasn't respected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1294962499


##########
docs/source/format/CDataInterface.rst:
##########
@@ -217,6 +217,8 @@ names and types of child fields are read from the child arrays.
 +------------------------+---------------------------------------------------+------------+
 | ``+us:I,J,...``        | sparse union with type ids I,J...                 |            |
 +------------------------+---------------------------------------------------+------------+
+| ``+r``                 | run-end encoded                                   |            |

Review Comment:
   It might be nice to include a brief summary of the structure of the children (I think this is done for the timestamp map types as footnotes).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1688930246

   @felipecrv I've pushed the Go implementation for REE with c-export/import


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1690638271

   @bkietz is out for now but it looks like both nits he had requested have been addressed. If there's no objections by tomorrow I'll merge this in.
   
   Thanks everyone!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1687856829

   @alippai @roee88 @tustvold Does the C Data Interface specification addition look fine here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade merged pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #37174:
URL: https://github.com/apache/arrow/pull/37174


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1295139769


##########
docs/source/format/CDataInterface.rst:
##########
@@ -217,6 +217,8 @@ names and types of child fields are read from the child arrays.
 +------------------------+---------------------------------------------------+------------+
 | ``+us:I,J,...``        | sparse union with type ids I,J...                 |            |
 +------------------------+---------------------------------------------------+------------+
+| ``+r``                 | run-end encoded                                   |            |

Review Comment:
   Hmmm. I mention the children in the examples section. A more detailed explanation would duplicate (badly) what is already said in the format specification pages. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297317340


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   But if you mean just for schema export (not the whole array round-trip), I can add any type. :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1297291851


##########
cpp/src/arrow/c/bridge_test.cc:
##########
@@ -443,6 +444,13 @@ TEST_F(TestSchemaExport, Union) {
              {ARROW_FLAG_NULLABLE});
 }
 
+TEST_F(TestSchemaExport, RunEndEncoded) {
+  TestNested(run_end_encoded(int16(), uint8()), {"+r", "s", "C"},
+             {"", "run_ends", "values"}, {ARROW_FLAG_NULLABLE, 0, ARROW_FLAG_NULLABLE});
+  TestNested(run_end_encoded(int64(), int32()), {"+r", "l", "i"},

Review Comment:
   The best I can do is a string array because `run_end_encode` doesn't produce REE-encoded arrays of nested values.
   
   https://github.com/apache/arrow/blob/main/cpp/src/arrow/compute/kernels/vector_run_end_encode.cc#L601-L621
   
   My reasoning for de-prioritizing that was that gains from run-end encoding complex objects (lists, structs) are less likely.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] felipecrv commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1296182882


##########
docs/source/format/CDataInterface.rst:
##########
@@ -217,6 +217,8 @@ names and types of child fields are read from the child arrays.
 +------------------------+---------------------------------------------------+------------+
 | ``+us:I,J,...``        | sparse union with type ids I,J...                 |            |
 +------------------------+---------------------------------------------------+------------+
+| ``+r``                 | run-end encoded                                   |            |

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #37174: GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37174:
URL: https://github.com/apache/arrow/pull/37174#discussion_r1300308212


##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1601,6 +1621,23 @@ struct ArrayImporter {
     return Status::OK();
   }
 
+  Status Visit(const RunEndEncodedType& type) {
+    RETURN_NOT_OK(CheckNumChildren(2));
+    if (c_struct_->n_buffers == 1) {
+      // REE arrays are not required to have any buffer,
+      // but an empty validity bitmap buffer is allowed.
+      RETURN_NOT_OK(AllocateArrayData());

Review Comment:
   We should distinguish between what we're doing internally in Arrow C++ and what we export/import using the C Data Interface. I'm not sure accepting a buffer for REE arrays makes sense if we don't export such a buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37174: GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37174:
URL: https://github.com/apache/arrow/pull/37174#issuecomment-1693997850

   After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 9958896fe067dbda351b9937cc97686c9d9ac93c.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/16228064206) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org