You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/29 19:41:10 UTC

[GitHub] [arrow] fatemehp opened a new pull request, #13997: Clean up Skip method naming rows->values

fatemehp opened a new pull request, #13997:
URL: https://github.com/apache/arrow/pull/13997

   The Skip method is skipping values and not rows. "values" is used throughout the code interchangeably with levels. Repeated fields may have multiple values, thus the use of "rows" is not accurate because we are not skipping over the values from the repeated field to the next row.
   
   Similarly, two other variables total_num_rows_ and seen_num_rows_ actually refer to values and not rows. So I updated them as well.
   
   I will add more tests for the Skip method that will clarify this behavior for repeated fields in a separate change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou merged pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
pitrou merged PR #13997:
URL: https://github.com/apache/arrow/pull/13997


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
emkornfield commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1230911260

   Actually it looks like the lint error is related to this change, lets fix that first
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] fatemehp commented on a diff in pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
fatemehp commented on code in PR #13997:
URL: https://github.com/apache/arrow/pull/13997#discussion_r958690748


##########
cpp/src/parquet/column_reader.h:
##########
@@ -218,9 +218,11 @@ class TypedColumnReader : public ColumnReader {
                                   int64_t valid_bits_offset, int64_t* levels_read,
                                   int64_t* values_read, int64_t* null_count) = 0;
 
-  // Skip reading levels
-  // Returns the number of levels skipped
-  virtual int64_t Skip(int64_t num_rows_to_skip) = 0;
+  // Skip reading values.
+  // Returns the number of values skipped.
+  // This function will NOT skip rows, and repeated fields may have multiple values
+  // corresponding to the same row.

Review Comment:
   I clarified the comment, please take a look and see if it makes it clear. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
emkornfield commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1230910113

   LGTM, I'll merge in a few hours unless anybody has more feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
emkornfield commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1232304660

   @jonkeane @thisisnic it looks like the R failure are unrelated to this PR, but wanted to double check before merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1232606084

   Thank you for contributing this @fatemehp. PR merged now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13997:
URL: https://github.com/apache/arrow/pull/13997#discussion_r958694116


##########
cpp/src/parquet/column_reader.h:
##########
@@ -218,10 +218,13 @@ class TypedColumnReader : public ColumnReader {
                                   int64_t valid_bits_offset, int64_t* levels_read,
                                   int64_t* values_read, int64_t* null_count) = 0;
 
-  // Skip reading values.
+  // Skip reading values. This method will work for both repeated and
+  // non-repeated fields. Note that this method is skipping values and not
+  // records. This distinction is important for repeated fields, meaning that
+  // we are not skipping over the values to the next record. We are skipping
+  // through them. So after the skip the iterator could be in the middle of a

Review Comment:
   Thanks for the update. This reads better, but I'm afraid the distinction between "skipping over" and "skipping through" is not clear to me. Could you perhaps try to explain better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] thisisnic commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
thisisnic commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1232536734

   @emkornfield Correct, that one is are unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1232816375

   Benchmark runs are scheduled for baseline = 8f071be75c5541aecf07fb6e1a0f91adb925efb2 and contender = c7e58cac918e672c2cb81be2b238e62c9e6de7a9. c7e58cac918e672c2cb81be2b238e62c9e6de7a9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:25.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5f85676f80374dfba248184544746c9a...273ea15499db4ac5b81e6a33c5b032af/)
   [Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/488ae555fb304b70b2e2830b3b91a0ee...0611870cb7f04e18b040a0008c5211d3/)
   [Failed :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/a579d697ca17409b9ab671008a494bde...ffb23939a4f24ae9a2fa6e9ab1cd9515/)
   [Finished :arrow_down:0.35% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17fc48a4972e491b962479f67b02cec2...dcadc73a8d7d48f3ad872dbe2d9b9aa5/)
   Buildkite builds:
   [Finished] [`c7e58cac` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1393)
   [Failed] [`c7e58cac` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1409)
   [Failed] [`c7e58cac` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1391)
   [Finished] [`c7e58cac` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1408)
   [Finished] [`8f071be7` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1392)
   [Failed] [`8f071be7` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1408)
   [Failed] [`8f071be7` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1390)
   [Finished] [`8f071be7` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1407)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13997:
URL: https://github.com/apache/arrow/pull/13997#discussion_r958502465


##########
cpp/src/parquet/column_reader.h:
##########
@@ -218,9 +218,11 @@ class TypedColumnReader : public ColumnReader {
                                   int64_t valid_bits_offset, int64_t* levels_read,
                                   int64_t* values_read, int64_t* null_count) = 0;
 
-  // Skip reading levels
-  // Returns the number of levels skipped
-  virtual int64_t Skip(int64_t num_rows_to_skip) = 0;
+  // Skip reading values.
+  // Returns the number of values skipped.
+  // This function will NOT skip rows, and repeated fields may have multiple values
+  // corresponding to the same row.

Review Comment:
   I find this comment confusing. Let's say I ask to skip 20 values:
   * If I am in a non-repeated field, nothing is skipped and 0 is returned?
   * If I am in a repeated field and the current row has 10 values, 10 is returned? How do I skip several rows?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13997: Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1230780548

   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
   
   Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] fatemehp commented on a diff in pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
fatemehp commented on code in PR #13997:
URL: https://github.com/apache/arrow/pull/13997#discussion_r958771016


##########
cpp/src/parquet/column_reader.h:
##########
@@ -218,10 +218,13 @@ class TypedColumnReader : public ColumnReader {
                                   int64_t valid_bits_offset, int64_t* levels_read,
                                   int64_t* values_read, int64_t* null_count) = 0;
 
-  // Skip reading values.
+  // Skip reading values. This method will work for both repeated and
+  // non-repeated fields. Note that this method is skipping values and not
+  // records. This distinction is important for repeated fields, meaning that
+  // we are not skipping over the values to the next record. We are skipping
+  // through them. So after the skip the iterator could be in the middle of a

Review Comment:
   I further clarified the comment with an example, please take a look.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #13997: MINOR: [PARQUET] Clean up Skip method naming rows->values

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13997:
URL: https://github.com/apache/arrow/pull/13997#issuecomment-1232816631

   ['Python', 'R'] benchmarks have high level of regressions.
   [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5f85676f80374dfba248184544746c9a...273ea15499db4ac5b81e6a33c5b032af/)
   [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/a579d697ca17409b9ab671008a494bde...ffb23939a4f24ae9a2fa6e9ab1cd9515/)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org