You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/06 13:53:55 UTC

[GitHub] [arrow] jvanstraten opened a new pull request, #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

jvanstraten opened a new pull request, #13528:
URL: https://github.com/apache/arrow/pull/13528

   A Substrait ProjectRel *appends* columns to the dataset, while Acero's project node replaces them (emit clauses are instead used to remove or swizzle columns). This PR prefixes the current columns in the project node to make the two compatible.
   
   I don't think a declaration includes information about the number of columns it generates, so I had to refactor a little bit to have relation ToProto return a struct of the declaration and the number of columns, in order to know how many columns to replicate. I expect this struct to grow in importance and features when ARROW-16986 is addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jvanstraten commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
jvanstraten commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r918390553


##########
cpp/src/arrow/engine/substrait/relation_internal.cc:
##########
@@ -215,16 +220,24 @@ Result<compute::Declaration> FromProto(const substrait::Rel& rel,
       }
       ARROW_ASSIGN_OR_RAISE(auto input, FromProto(project.input(), ext_set));
 
+      // NOTE: Substrait ProjectRels *append* columns, while Acero's project node replaces
+      // them. Therefore, we need to prefix all the current columns for compatibility.
       std::vector<compute::Expression> expressions;

Review Comment:
   96f8c93



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r918626859


##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   @jvanstraten Right, it's basically the convention used accross Arrow. That said, I do not really find it weird because frequently you'll act on sizes by adding/removing offsets from them - and those offsets are usually signed. It makes sense to make all integers signed, IMHO.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13528:
URL: https://github.com/apache/arrow/pull/13528#issuecomment-1176252100

   https://issues.apache.org/jira/browse/ARROW-16989


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou merged pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
pitrou merged PR #13528:
URL: https://github.com/apache/arrow/pull/13528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
vibhatha commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r918461211


##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   For my curiosity: 
   
   I understand your point. I am not an expert on the exact usage, although I assumed the current customs were adopted based on a certain logic. But for better understanding I looked into CUDF which require to expose a similar attribute as follows.
   
   https://github.com/pytorch/pytorch/blob/5f8c2076dfd834b21cfafe2f35d19c3d02a958dc/caffe2/core/tensor.h#L386
   
   Here they use size_type (I guess that is the same as size_t). 
   
   PS: Numpy and PyTorch both represents the dimension in `int` or `int64_t` but I guess it can be negative according to some definitions (I am not entirely clear on the possibility).
   
   cc @pitrou @wesm 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jvanstraten commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
jvanstraten commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r918390405


##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   96f8c93



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jvanstraten commented on pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
jvanstraten commented on PR #13528:
URL: https://github.com/apache/arrow/pull/13528#issuecomment-1180915893

   I didn't add a test because none of the existing tests were anywhere near that detail (they all just round-trip, there is no functional testing yet). IMO it makes more sense to set up the framework for this in a separate PR. I don't know their github handle, but I believe Richard Tia will be working on that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jvanstraten commented on pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
jvanstraten commented on PR #13528:
URL: https://github.com/apache/arrow/pull/13528#issuecomment-1176499772

   I'm not sure what's going on with CI. I don't think it's something I did?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r917912040


##########
cpp/src/arrow/engine/substrait/relation_internal.cc:
##########
@@ -215,16 +220,24 @@ Result<compute::Declaration> FromProto(const substrait::Rel& rel,
       }
       ARROW_ASSIGN_OR_RAISE(auto input, FromProto(project.input(), ext_set));
 
+      // NOTE: Substrait ProjectRels *append* columns, while Acero's project node replaces
+      // them. Therefore, we need to prefix all the current columns for compatibility.
       std::vector<compute::Expression> expressions;

Review Comment:
   May want to presize this?



##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   Right, this can be an `int`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
vibhatha commented on PR #13528:
URL: https://github.com/apache/arrow/pull/13528#issuecomment-1181202804

   > I didn't add a test because none of the existing tests were anywhere near that detail (they all just round-trip, there is no functional testing yet). IMO it makes more sense to set up the framework for this in a separate PR. I don't know their github handle, but I believe Richard Tia will be working on that.
   
   cc @pitrou There is another effort to finalize the functional end-to-end tests in Python.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
vibhatha commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r916656040


##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   Just curious, should this be just `int` or `size_t`. The `Table` and `RecordBatch` interfaces provide `num_columns` as int. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
vibhatha commented on PR #13528:
URL: https://github.com/apache/arrow/pull/13528#issuecomment-1178779195

   > I'm not sure what's going on with CI. I don't think it's something I did?
   
   I experienced the same with the python tests. It could be flaky. But not exaclty sure thought. 
   
   May be another push and see if it is going to repeat. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jvanstraten commented on a diff in pull request #13528: ARROW-16989: [C++] Substrait ProjectRel is interpreted incorrectly

Posted by GitBox <gi...@apache.org>.
jvanstraten commented on code in PR #13528:
URL: https://github.com/apache/arrow/pull/13528#discussion_r918384419


##########
cpp/src/arrow/engine/substrait/relation_internal.h:
##########
@@ -30,8 +30,17 @@
 namespace arrow {
 namespace engine {
 
+/// Information resulting from converting a Substrait relation.
+struct DeclarationInfo {
+  /// The compute declaration produced thus far.
+  compute::Declaration declaration;
+
+  /// The number of columns returned by the declaration.
+  size_t num_columns;

Review Comment:
   Seems weird to me to use a signed integer for a size; IMO `Table` and `RecordBatch` are wrong to do that (if the 32/64-bitness is an issue, at least use `uint32_t` or `unsigned int`). Now you'd always get warnings when comparing with the `size_t`'s returned by STL containers.
   
   That said, I guess I won't argue with precedence...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org