You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/21 22:32:58 UTC
[GitHub] [spark] sunchao opened a new pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
sunchao opened a new pull request #33006:
URL: https://github.com/apache/spark/pull/33006
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Move all the bookkeeping states while scanning a Parquet column chunk into a single class `ParquetReadState`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
As suggested [here](https://github.com/apache/spark/pull/32753#discussion_r655580942). To support column index in the vectorized reader path, we'll going to introduce more states to track. These are spread across different classes which make the code harder to maintain. Therefore, this proposes to move them into a single class so they can be managed better.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655896700
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
Review comment:
Can we let `readPage` to return the value count? then we don't need the state `pageValueCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656404799
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
Review comment:
Shall we have an initial value, `-1`, and use it in`advanceOffset` to validate `resetForBatch` is called properly before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865717921
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865438711
@dongjoon-hyun nope, this just replicate the existing logic - the `newOffset - offset < valuesToReadInBatch` here serves as an extra check so that we'll never read more than specified in the batch size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865435760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #33006:
URL: https://github.com/apache/spark/pull/33006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865718065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656401983
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
+
+ /** The remaining number of values to read in the current page */
+ int valuesToReadInPage;
+
+ /** The remaining number of values to read in the current batch */
+ int valuesToReadInBatch;
Review comment:
Just a question. Previously, we tracked with `long` type for `valuesRead` and the others. After this PR, `int` is enough and this is aligned with Parquet file definition. Did I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865909419
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656508834
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
Review comment:
That's somewhat useful but it adds a branching in a hot path. Also we may have to remember reset the offset to -1 after reading a batch. Because these are internal methods, I guess it's not easy to go wrong without being captured by unit tests and other things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655896169
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
looks OK, it's just a few CPU cycles.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
Review comment:
Can we let `readPage` to return the value count? then we don't need the state `pageValueCount`.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
This looks inefficient. Can we follow the old code and calculate the `left` first, then do `while (left > 0)`?
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
e.g.
```
int offset = state.offset;
int left = Math.min(state.valuesToReadInPage, state. valuesToReadInBatch)
while (left > 0) {
...
offset += n;
left -= n;
}
```
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -216,53 +195,49 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
boolean needTransform = castLongToInt || isUnsignedInt32 || isUnsignedInt64;
column.setDictionary(new ParquetDictionary(dictionary, needTransform));
} else {
- updater.decodeDictionaryIds(num, rowId, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset - startOffset, startOffset, column,
+ dictionaryIds, dictionary);
}
} else {
- if (column.hasDictionary() && rowId != 0) {
+ if (column.hasDictionary() && readState.offset != 0) {
// This batch already has dictionary encoded values but this new page is not. The batch
// does not support a mix of dictionary and not so we will decode the dictionary.
- updater.decodeDictionaryIds(rowId, 0, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset, 0, column, dictionaryIds, dictionary);
}
column.setDictionary(null);
VectorizedValuesReader valuesReader = (VectorizedValuesReader) dataColumn;
- defColumn.readBatch(num, rowId, column, maxDefLevel, valuesReader, updater);
+ defColumn.readBatch(readState, column, valuesReader, updater);
}
-
- valuesRead += num;
- rowId += num;
- total -= num;
}
}
- private void readPage() {
+ private int readPage() {
DataPage page = pageReader.readPage();
- // TODO: Why is this a visitor?
- page.accept(new DataPage.Visitor<Void>() {
+ return page.accept(new DataPage.Visitor<Integer>() {
@Override
- public Void visit(DataPageV1 dataPageV1) {
+ public Integer visit(DataPageV1 dataPageV1) {
Review comment:
To avoid boxing, we can assign the valueCount to a local variale and return it, e.g.
```
int pageValueCount = 0;
page.accept(new DataPage.Visitor<Integer>() {
...
pageValueCount = readPageV1(dataPageV1);
return null;
...
}
return pageValueCount;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865718065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865389146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655935518
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
Review comment:
Yes, let me change it. We can also get rid of `totalValueCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866406464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656942709
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -170,27 +170,28 @@ public int readInteger() {
* }
*/
public void readBatch(
- int total,
- int offset,
+ ParquetReadState state,
WritableColumnVector values,
- int maxDefinitionLevel,
VectorizedValuesReader valueReader,
ParquetVectorUpdater updater) throws IOException {
Review comment:
For this one, let's proceed separately if needed. There might be more places like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866405766
**[Test build #140163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140163/testReport)** for PR 33006 at commit [`f5e97b7`](https://github.com/apache/spark/commit/f5e97b77005456894cce6e01bc54160ee0cf1d01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865409035
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656785038
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -170,27 +170,28 @@ public int readInteger() {
* }
*/
public void readBatch(
- int total,
- int offset,
+ ParquetReadState state,
WritableColumnVector values,
- int maxDefinitionLevel,
VectorizedValuesReader valueReader,
ParquetVectorUpdater updater) throws IOException {
Review comment:
good catch - I think there isn't `IOException` to throw here and we can just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865705093
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656777628
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -216,53 +195,49 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
boolean needTransform = castLongToInt || isUnsignedInt32 || isUnsignedInt64;
column.setDictionary(new ParquetDictionary(dictionary, needTransform));
} else {
- updater.decodeDictionaryIds(num, rowId, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset - startOffset, startOffset, column,
+ dictionaryIds, dictionary);
}
} else {
- if (column.hasDictionary() && rowId != 0) {
+ if (column.hasDictionary() && readState.offset != 0) {
// This batch already has dictionary encoded values but this new page is not. The batch
// does not support a mix of dictionary and not so we will decode the dictionary.
- updater.decodeDictionaryIds(rowId, 0, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset, 0, column, dictionaryIds, dictionary);
}
column.setDictionary(null);
VectorizedValuesReader valuesReader = (VectorizedValuesReader) dataColumn;
- defColumn.readBatch(num, rowId, column, maxDefLevel, valuesReader, updater);
+ defColumn.readBatch(readState, column, valuesReader, updater);
}
-
- valuesRead += num;
- rowId += num;
- total -= num;
}
}
- private void readPage() {
+ private int readPage() {
DataPage page = pageReader.readPage();
- // TODO: Why is this a visitor?
- page.accept(new DataPage.Visitor<Void>() {
+ return page.accept(new DataPage.Visitor<Integer>() {
@Override
- public Void visit(DataPageV1 dataPageV1) {
+ public Integer visit(DataPageV1 dataPageV1) {
Review comment:
ah I see, let's leave it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656399691
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
Review comment:
`final int maxDefinitionLevel`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866272842
**[Test build #140163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140163/testReport)** for PR 33006 at commit [`f5e97b7`](https://github.com/apache/spark/commit/f5e97b77005456894cce6e01bc54160ee0cf1d01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866310918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44689/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865389146
**[Test build #140103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140103/testReport)** for PR 33006 at commit [`b86824a`](https://github.com/apache/spark/commit/b86824a7e4c7efa4d685e4d15554b881a4d83300).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865438711
@dongjoon-hyun nope, this just replicate the existing logic - the `newOffset - offset < valuesToReadInBatch` here serves as an extra check so that we'll never read more than specified in the batch size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866406464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865499185
I think this makes codes easier to read. cc @michal-databricks @cloud-fan FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655914360
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
Thanks. I think we can just do:
```java
if (column.hasDictionary() || (startOffset == 0 && isLazyDecodingSupported(typeName)))
```
and it should have the same effect.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
I don't have strong opinion on this. It's just that in the column index PR we'll introduce `rowId` and instead of `left` we'll need something like `valuesLeft` and `rowsLeft`. In addition there'll be several branches where we need to update the `rowsLeft`, so it'll be something like:
```java
int offset = state.offset;
long rowId = state.rowId;
int valuesLeft = state.valuesToReadInBatch;
int rowsLeft = state.valuesToReadInPage;
while (valuesLeft > 0 && rowsLeft > 0) {
if (...) {
rowId += n;
rowsLeft -= n;
} else {
...
if (toSkip > 0) {
...
rowId += n;
rowsLeft -= n;
}
...
offset += n;
valuesLeft -= n;
rowId += n;
rowsLeft -= n;
}
}
```
so slightly more complicated.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
Review comment:
Yes, let me change it. We can also get rid of `totalValueCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865435547
**[Test build #140103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140103/testReport)** for PR 33006 at commit [`b86824a`](https://github.com/apache/spark/commit/b86824a7e4c7efa4d685e4d15554b881a4d83300).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `final class ParquetReadState `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866310918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44689/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865389146
**[Test build #140103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140103/testReport)** for PR 33006 at commit [`b86824a`](https://github.com/apache/spark/commit/b86824a7e4c7efa4d685e4d15554b881a4d83300).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865409035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656404799
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
Review comment:
Shall we have an initial value, `-1` and use it in`advanceOffset` to validate `resetForBatch` is called properly before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656420130
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -216,53 +195,49 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
boolean needTransform = castLongToInt || isUnsignedInt32 || isUnsignedInt64;
column.setDictionary(new ParquetDictionary(dictionary, needTransform));
} else {
- updater.decodeDictionaryIds(num, rowId, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset - startOffset, startOffset, column,
+ dictionaryIds, dictionary);
}
} else {
- if (column.hasDictionary() && rowId != 0) {
+ if (column.hasDictionary() && readState.offset != 0) {
// This batch already has dictionary encoded values but this new page is not. The batch
// does not support a mix of dictionary and not so we will decode the dictionary.
- updater.decodeDictionaryIds(rowId, 0, column, dictionaryIds, dictionary);
+ updater.decodeDictionaryIds(readState.offset, 0, column, dictionaryIds, dictionary);
}
column.setDictionary(null);
VectorizedValuesReader valuesReader = (VectorizedValuesReader) dataColumn;
- defColumn.readBatch(num, rowId, column, maxDefLevel, valuesReader, updater);
+ defColumn.readBatch(readState, column, valuesReader, updater);
}
-
- valuesRead += num;
- rowId += num;
- total -= num;
}
}
- private void readPage() {
+ private int readPage() {
DataPage page = pageReader.readPage();
- // TODO: Why is this a visitor?
- page.accept(new DataPage.Visitor<Void>() {
+ return page.accept(new DataPage.Visitor<Integer>() {
@Override
- public Void visit(DataPageV1 dataPageV1) {
+ public Integer visit(DataPageV1 dataPageV1) {
Review comment:
I'm not sure you can do that, since the variable has to be final in order to be accessed within inner class.
Also it probably doesn't matter that much since this is not the hot path (I'm very tempted to remove the visitor pattern here though...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655897530
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
This looks inefficient. Can we follow the old code and calculate the `left` first, then do `while (left > 0)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865389146
**[Test build #140103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140103/testReport)** for PR 33006 at commit [`b86824a`](https://github.com/apache/spark/commit/b86824a7e4c7efa4d685e4d15554b881a4d83300).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865405309
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865435760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865908034
**[Test build #140124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140124/testReport)** for PR 33006 at commit [`e11d20e`](https://github.com/apache/spark/commit/e11d20e679d45c255a3269e4256808f657ed105c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865499185
I think this makes codes easier to read. cc @michal-databricks @cloud-fan FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655899492
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
e.g.
```
int offset = state.offset;
int left = Math.min(state.valuesToReadInPage, state. valuesToReadInBatch)
while (left > 0) {
...
offset += n;
left -= n;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865409035
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865668910
**[Test build #140124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140124/testReport)** for PR 33006 at commit [`e11d20e`](https://github.com/apache/spark/commit/e11d20e679d45c255a3269e4256808f657ed105c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865408880
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656409834
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
+
+ /** The remaining number of values to read in the current page */
+ int valuesToReadInPage;
+
+ /** The remaining number of values to read in the current batch */
+ int valuesToReadInBatch;
Review comment:
Yes. Previously `valuesRead` is the number of values read so far within the column chunk, so it has to be `long`. However, it is not that useful. Here we changed to only track values read so far within a page and within a single batch, both of which are integers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656401983
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+/**
+ * Helper class to store intermediate state while reading a Parquet column chunk.
+ */
+final class ParquetReadState {
+ /** Maximum definition level */
+ int maxDefinitionLevel;
+
+ /** The offset to add the next value in the current batch */
+ int offset;
+
+ /** The remaining number of values to read in the current page */
+ int valuesToReadInPage;
+
+ /** The remaining number of values to read in the current batch */
+ int valuesToReadInBatch;
Review comment:
Just a question. Previously, we tracked with `long` type. After this PR, `int` is enough and this is aligned with Parquet file definition. Did I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655914360
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
Thanks. I think we can just do:
```java
if (column.hasDictionary() || (startOffset == 0 && isLazyDecodingSupported(typeName)))
```
and it should have the same effect.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -199,48 +200,51 @@ public void readBatch(
break;
}
offset += n;
- left -= n;
currentCount -= n;
}
+
+ state.advanceOffset(offset);
}
/**
* Decoding for dictionary ids. The IDs are populated into `values` and the nullability is
* populated into `nulls`.
*/
public void readIntegers(
- int total,
+ ParquetReadState state,
WritableColumnVector values,
WritableColumnVector nulls,
- int rowId,
- int level,
VectorizedValuesReader data) throws IOException {
- int left = total;
- while (left > 0) {
+ int offset = state.offset;
+
+ while (state.hasMoreInPage(offset)) {
Review comment:
I don't have strong opinion on this. It's just that in the column index PR we'll introduce `rowId` and instead of `left` we'll need something like `valuesLeft` and `rowsLeft`. In addition there'll be several branches where we need to update the `rowsLeft`, so it'll be something like:
```java
int offset = state.offset;
long rowId = state.rowId;
int valuesLeft = state.valuesToReadInBatch;
int rowsLeft = state.valuesToReadInPage;
while (valuesLeft > 0 && rowsLeft > 0) {
if (...) {
rowId += n;
rowsLeft -= n;
} else {
...
if (toSkip > 0) {
...
rowId += n;
rowsLeft -= n;
}
...
offset += n;
valuesLeft -= n;
rowId += n;
rowsLeft -= n;
}
}
```
so slightly more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655896169
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
looks OK, it's just a few CPU cycles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655847305
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
Just to make sure we don't miss anything, there's a small side effect here. Previously `(rowId == 0 && isLazyDecodingSupported(typeName))` wasn't executed when `column.hasDictionary()` true, but now it's executed alway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r656661993
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -170,27 +170,28 @@ public int readInteger() {
* }
*/
public void readBatch(
- int total,
- int offset,
+ ParquetReadState state,
WritableColumnVector values,
- int maxDefinitionLevel,
VectorizedValuesReader valueReader,
ParquetVectorUpdater updater) throws IOException {
Review comment:
not related to this PR: is there an `IOException` to throw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866272842
**[Test build #140163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140163/testReport)** for PR 33006 at commit [`f5e97b7`](https://github.com/apache/spark/commit/f5e97b77005456894cce6e01bc54160ee0cf1d01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33006:
URL: https://github.com/apache/spark/pull/33006#discussion_r655847305
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
##########
@@ -174,24 +162,29 @@ void readBatch(int total, WritableColumnVector column) throws IOException {
// page.
dictionaryIds = column.reserveDictionaryIds(total);
}
- while (total > 0) {
+ readState.resetForBatch(total);
+ while (readState.valuesToReadInBatch > 0) {
// Compute the number of values we want to read in this page.
- int leftInPage = (int) (endOfPageValueCount - valuesRead);
- if (leftInPage == 0) {
+ if (readState.valuesToReadInPage == 0) {
readPage();
- leftInPage = (int) (endOfPageValueCount - valuesRead);
+ readState.resetForPage(pageValueCount);
}
- int num = Math.min(total, leftInPage);
PrimitiveType.PrimitiveTypeName typeName =
descriptor.getPrimitiveType().getPrimitiveTypeName();
if (isCurrentPageDictionaryEncoded) {
+ boolean supportLazyDecoding = readState.offset == 0 &&
+ isLazyDecodingSupported(typeName);
+
+ // Save starting offset in case we need to decode dictionary IDs.
+ int startOffset = readState.offset;
+
// Read and decode dictionary ids.
- defColumn.readIntegers(
- num, dictionaryIds, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
+ defColumn.readIntegers(readState, dictionaryIds, column,
+ (VectorizedValuesReader) dataColumn);
// TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process
// the values to add microseconds precision.
- if (column.hasDictionary() || (rowId == 0 && isLazyDecodingSupported(typeName))) {
+ if (column.hasDictionary() || supportLazyDecoding) {
Review comment:
Just to make sure we don't miss anything, there's a small side effect here. Previously `(rowId == 0 && isLazyDecodingSupported(typeName))` wasn't executed when `column.hasDictionary()` true, but now it's executed alway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865909419
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-866303187
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44689/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865668910
**[Test build #140124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140124/testReport)** for PR 33006 at commit [`e11d20e`](https://github.com/apache/spark/commit/e11d20e679d45c255a3269e4256808f657ed105c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33006: [SPARK-35846][SQL] Introduce ParquetReadState to track various states while reading a Parquet column chunk
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33006:
URL: https://github.com/apache/spark/pull/33006#issuecomment-865409035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org